Embed Size (px)
Citation preview
Windows Azure Storage Deep DiveJai HaridasSoftware EngineerMicrosoft Corporation
SESSION CODE COS310
AGENDA
Introduction to StorageStorage Abstractions
Blobs DrivesTablesQueues
SummaryTribune CompanyQuestions
WINDOWS AZURE STORAGE
Enables developers apps and users to access cloud storageScalable durable and availableAnywhere at anytime accessOnly pay for what the service uses
Easy to use REST and Client LibrariesBlobs tables queues
Existing NTFS APIsDrives
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE STORAGE ACCOUNT
User creates a globally unique storage account nameCan choose geo-location to host storage account
US ndash ldquoNorth Centralrdquo and ldquoSouth CentralrdquoEurope ndash ldquoNorthrdquo and ldquoWestrdquoAsia ndash ldquoEastrdquo and ldquoSoutheastrdquo
Can co-locate storage account with compute accountReceive a 512 bit secret key when creating account
You get two keys which can be used to rotate
BLOB STORAGE CONCEPTS
BlobContainerAccount
cohowinery
images
PIC01JPG
PIC02JPG
videos VID1AVI
httpltaccountgtblobcorewindowsnetltcontainergtltblobnamegt
BLOB FEATURES AND FUNCTIONS
Programming InterfacesPutBlob
Insert blob and Update blobGetBlob
Get whole blob or a specific rangeDeleteBlobCopyBlobSnapshotBlobLeaseBlob
Associate Metadata with BlobStandard HTTP metadata (Cache-Control Content-Encoding Content-Type etc)Metadata is ltname valuegt pairs up to 8KB per blob
BLOB CLIENT LIBRARY EXAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudBlobClient blobClient = new CloudBlobClient(
accountBlobEndpoint accountCredentials) Create ContainerCloudBlobContainer cloudContainer = blobClientGetContainerReference(containerName)bool hasCreated = cloudContainerCreateIfNotExist()
Access Blob in the ContainerCloudBlob cloudBlob = cloudContainerGetBlobReference(blobName)
BlobRequestOptions has retry policy timeout etcBlobRequestOptions options = new BlobRequestOptions() Upload the local file to Blob servicecloudBlobUploadFile(uploadFileName options)
Download to local file namecloudBlobDownloadToFile(downloadFileName options)
TWO TYPES OF BLOBS UNDER THE HOOD
Block Blob Targeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedSize limit 200GB per blob
Page BlobTargeted at random readwrite workloadsEach blob consists of an array of pages
Each page range write is committed on PUTSize limit 1TB per blob
BLOCK BLOBStreaming Workload w Random Reads + Committed Writes
Uploading a large blob
10 GB Movie
Windows Azure Storage
Bloc
k Id
1Bl
ock
Id 2
Bloc
k Id
3
Bloc
k Id
N
blobName = ldquoblobwmvrdquoPutBlock(blobName blockId1 block1Bits)PutBlock(blobName blockId2 block2Bits)helliphelliphelliphellipPutBlock(blobName blockIdN blockNBits)PutBlockList(blobName
blockId1 blockId2hellipblockIdN)
blobwmvblobwmv
BenefitUpdate the blob via blocks any way you wantEfficient continuation and retryParallel and out of order upload of blocks
PAGE BLOB ndash RANDOM READWRITECreate MyBlob
Specify Blob Size = 10 GB
Fixed Page Size = 512 bytes
Random Access OperationsPutPage[512 2048)PutPage[0 1024)ClearPage[512 1536)PutPage[20482560)
GetPageRange[0 4096) returns valid data ranges
[0512) [15362560)
GetBlob[1000 2048) returnsAll 0 for first 536 bytesNext 512 bytes are data stored in [15362048)
0
10 GB
10 GB Address Space
512
1024
1536
2048
2560
WINDOWS AZURE CONTENT DELIVERY NETWORK
ScenarioFrequently accessed blobsBlobs accessed from many geographic locations
Windows Azure Content Delivery Network (CDN)Cache and serve your Windows Azure Blobs from our network18 locations globally (US Europe Asia Australia and South America) and growing
BenefitBetter experience for users far from blob serviceProvide high-bandwidth content delivery around the world for popular events
WINDOWS CDN DETAILS
Registering for CDN in Windows Azure Dev Portal Enable storage account to use CDNGet back a domain name and can specify your own custom domain name to use
Windows Azure CDN URL httpltidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom Windows Azure Storage URL httpaccountblobcorewindowsnet
Using Windows Azure CDN for Blob AccessSpecify blobs HTTP Cache-Control policy as TTLMake blobrsquos container(s) publically accessible Reference blobs using CDN domain name
The request is routed to our closest edge locationOn miss or TTL expiration blobs are retrieved from blob service and cached in the CDNAllows the blobs to be served from our CDN caches
SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs but do not want to distribute their secret key
Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs
httpscohowineryblobcorewindowsnetimagespic1jpg
Valid time rangest=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
BLOB TIPSFor high throughput clients
Set ServicePointManagerDefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2UploadDownload multiple files in parallelParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size gt= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KBExpectedThroughputInKBps
Use retry and exponential backoff for timeoutsserver busy
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
AGENDA
Introduction to StorageStorage Abstractions
Blobs DrivesTablesQueues
SummaryTribune CompanyQuestions
WINDOWS AZURE STORAGE
Enables developers apps and users to access cloud storageScalable durable and availableAnywhere at anytime accessOnly pay for what the service uses
Easy to use REST and Client LibrariesBlobs tables queues
Existing NTFS APIsDrives
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE STORAGE ACCOUNT
User creates a globally unique storage account nameCan choose geo-location to host storage account
US ndash ldquoNorth Centralrdquo and ldquoSouth CentralrdquoEurope ndash ldquoNorthrdquo and ldquoWestrdquoAsia ndash ldquoEastrdquo and ldquoSoutheastrdquo
Can co-locate storage account with compute accountReceive a 512 bit secret key when creating account
You get two keys which can be used to rotate
BLOB STORAGE CONCEPTS
BlobContainerAccount
cohowinery
images
PIC01JPG
PIC02JPG
videos VID1AVI
httpltaccountgtblobcorewindowsnetltcontainergtltblobnamegt
BLOB FEATURES AND FUNCTIONS
Programming InterfacesPutBlob
Insert blob and Update blobGetBlob
Get whole blob or a specific rangeDeleteBlobCopyBlobSnapshotBlobLeaseBlob
Associate Metadata with BlobStandard HTTP metadata (Cache-Control Content-Encoding Content-Type etc)Metadata is ltname valuegt pairs up to 8KB per blob
BLOB CLIENT LIBRARY EXAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudBlobClient blobClient = new CloudBlobClient(
accountBlobEndpoint accountCredentials) Create ContainerCloudBlobContainer cloudContainer = blobClientGetContainerReference(containerName)bool hasCreated = cloudContainerCreateIfNotExist()
Access Blob in the ContainerCloudBlob cloudBlob = cloudContainerGetBlobReference(blobName)
BlobRequestOptions has retry policy timeout etcBlobRequestOptions options = new BlobRequestOptions() Upload the local file to Blob servicecloudBlobUploadFile(uploadFileName options)
Download to local file namecloudBlobDownloadToFile(downloadFileName options)
TWO TYPES OF BLOBS UNDER THE HOOD
Block Blob Targeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedSize limit 200GB per blob
Page BlobTargeted at random readwrite workloadsEach blob consists of an array of pages
Each page range write is committed on PUTSize limit 1TB per blob
BLOCK BLOBStreaming Workload w Random Reads + Committed Writes
Uploading a large blob
10 GB Movie
Windows Azure Storage
Bloc
k Id
1Bl
ock
Id 2
Bloc
k Id
3
Bloc
k Id
N
blobName = ldquoblobwmvrdquoPutBlock(blobName blockId1 block1Bits)PutBlock(blobName blockId2 block2Bits)helliphelliphelliphellipPutBlock(blobName blockIdN blockNBits)PutBlockList(blobName
blockId1 blockId2hellipblockIdN)
blobwmvblobwmv
BenefitUpdate the blob via blocks any way you wantEfficient continuation and retryParallel and out of order upload of blocks
PAGE BLOB ndash RANDOM READWRITECreate MyBlob
Specify Blob Size = 10 GB
Fixed Page Size = 512 bytes
Random Access OperationsPutPage[512 2048)PutPage[0 1024)ClearPage[512 1536)PutPage[20482560)
GetPageRange[0 4096) returns valid data ranges
[0512) [15362560)
GetBlob[1000 2048) returnsAll 0 for first 536 bytesNext 512 bytes are data stored in [15362048)
0
10 GB
10 GB Address Space
512
1024
1536
2048
2560
WINDOWS AZURE CONTENT DELIVERY NETWORK
ScenarioFrequently accessed blobsBlobs accessed from many geographic locations
Windows Azure Content Delivery Network (CDN)Cache and serve your Windows Azure Blobs from our network18 locations globally (US Europe Asia Australia and South America) and growing
BenefitBetter experience for users far from blob serviceProvide high-bandwidth content delivery around the world for popular events
WINDOWS CDN DETAILS
Registering for CDN in Windows Azure Dev Portal Enable storage account to use CDNGet back a domain name and can specify your own custom domain name to use
Windows Azure CDN URL httpltidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom Windows Azure Storage URL httpaccountblobcorewindowsnet
Using Windows Azure CDN for Blob AccessSpecify blobs HTTP Cache-Control policy as TTLMake blobrsquos container(s) publically accessible Reference blobs using CDN domain name
The request is routed to our closest edge locationOn miss or TTL expiration blobs are retrieved from blob service and cached in the CDNAllows the blobs to be served from our CDN caches
SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs but do not want to distribute their secret key
Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs
httpscohowineryblobcorewindowsnetimagespic1jpg
Valid time rangest=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
BLOB TIPSFor high throughput clients
Set ServicePointManagerDefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2UploadDownload multiple files in parallelParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size gt= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KBExpectedThroughputInKBps
Use retry and exponential backoff for timeoutsserver busy
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
WINDOWS AZURE STORAGE
Enables developers apps and users to access cloud storageScalable durable and availableAnywhere at anytime accessOnly pay for what the service uses
Easy to use REST and Client LibrariesBlobs tables queues
Existing NTFS APIsDrives
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE STORAGE ACCOUNT
User creates a globally unique storage account nameCan choose geo-location to host storage account
US ndash ldquoNorth Centralrdquo and ldquoSouth CentralrdquoEurope ndash ldquoNorthrdquo and ldquoWestrdquoAsia ndash ldquoEastrdquo and ldquoSoutheastrdquo
Can co-locate storage account with compute accountReceive a 512 bit secret key when creating account
You get two keys which can be used to rotate
BLOB STORAGE CONCEPTS
BlobContainerAccount
cohowinery
images
PIC01JPG
PIC02JPG
videos VID1AVI
httpltaccountgtblobcorewindowsnetltcontainergtltblobnamegt
BLOB FEATURES AND FUNCTIONS
Programming InterfacesPutBlob
Insert blob and Update blobGetBlob
Get whole blob or a specific rangeDeleteBlobCopyBlobSnapshotBlobLeaseBlob
Associate Metadata with BlobStandard HTTP metadata (Cache-Control Content-Encoding Content-Type etc)Metadata is ltname valuegt pairs up to 8KB per blob
BLOB CLIENT LIBRARY EXAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudBlobClient blobClient = new CloudBlobClient(
accountBlobEndpoint accountCredentials) Create ContainerCloudBlobContainer cloudContainer = blobClientGetContainerReference(containerName)bool hasCreated = cloudContainerCreateIfNotExist()
Access Blob in the ContainerCloudBlob cloudBlob = cloudContainerGetBlobReference(blobName)
BlobRequestOptions has retry policy timeout etcBlobRequestOptions options = new BlobRequestOptions() Upload the local file to Blob servicecloudBlobUploadFile(uploadFileName options)
Download to local file namecloudBlobDownloadToFile(downloadFileName options)
TWO TYPES OF BLOBS UNDER THE HOOD
Block Blob Targeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedSize limit 200GB per blob
Page BlobTargeted at random readwrite workloadsEach blob consists of an array of pages
Each page range write is committed on PUTSize limit 1TB per blob
BLOCK BLOBStreaming Workload w Random Reads + Committed Writes
Uploading a large blob
10 GB Movie
Windows Azure Storage
Bloc
k Id
1Bl
ock
Id 2
Bloc
k Id
3
Bloc
k Id
N
blobName = ldquoblobwmvrdquoPutBlock(blobName blockId1 block1Bits)PutBlock(blobName blockId2 block2Bits)helliphelliphelliphellipPutBlock(blobName blockIdN blockNBits)PutBlockList(blobName
blockId1 blockId2hellipblockIdN)
blobwmvblobwmv
BenefitUpdate the blob via blocks any way you wantEfficient continuation and retryParallel and out of order upload of blocks
PAGE BLOB ndash RANDOM READWRITECreate MyBlob
Specify Blob Size = 10 GB
Fixed Page Size = 512 bytes
Random Access OperationsPutPage[512 2048)PutPage[0 1024)ClearPage[512 1536)PutPage[20482560)
GetPageRange[0 4096) returns valid data ranges
[0512) [15362560)
GetBlob[1000 2048) returnsAll 0 for first 536 bytesNext 512 bytes are data stored in [15362048)
0
10 GB
10 GB Address Space
512
1024
1536
2048
2560
WINDOWS AZURE CONTENT DELIVERY NETWORK
ScenarioFrequently accessed blobsBlobs accessed from many geographic locations
Windows Azure Content Delivery Network (CDN)Cache and serve your Windows Azure Blobs from our network18 locations globally (US Europe Asia Australia and South America) and growing
BenefitBetter experience for users far from blob serviceProvide high-bandwidth content delivery around the world for popular events
WINDOWS CDN DETAILS
Registering for CDN in Windows Azure Dev Portal Enable storage account to use CDNGet back a domain name and can specify your own custom domain name to use
Windows Azure CDN URL httpltidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom Windows Azure Storage URL httpaccountblobcorewindowsnet
Using Windows Azure CDN for Blob AccessSpecify blobs HTTP Cache-Control policy as TTLMake blobrsquos container(s) publically accessible Reference blobs using CDN domain name
The request is routed to our closest edge locationOn miss or TTL expiration blobs are retrieved from blob service and cached in the CDNAllows the blobs to be served from our CDN caches
SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs but do not want to distribute their secret key
Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs
httpscohowineryblobcorewindowsnetimagespic1jpg
Valid time rangest=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
BLOB TIPSFor high throughput clients
Set ServicePointManagerDefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2UploadDownload multiple files in parallelParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size gt= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KBExpectedThroughputInKBps
Use retry and exponential backoff for timeoutsserver busy
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE STORAGE ACCOUNT
User creates a globally unique storage account nameCan choose geo-location to host storage account
US ndash ldquoNorth Centralrdquo and ldquoSouth CentralrdquoEurope ndash ldquoNorthrdquo and ldquoWestrdquoAsia ndash ldquoEastrdquo and ldquoSoutheastrdquo
Can co-locate storage account with compute accountReceive a 512 bit secret key when creating account
You get two keys which can be used to rotate
BLOB STORAGE CONCEPTS
BlobContainerAccount
cohowinery
images
PIC01JPG
PIC02JPG
videos VID1AVI
httpltaccountgtblobcorewindowsnetltcontainergtltblobnamegt
BLOB FEATURES AND FUNCTIONS
Programming InterfacesPutBlob
Insert blob and Update blobGetBlob
Get whole blob or a specific rangeDeleteBlobCopyBlobSnapshotBlobLeaseBlob
Associate Metadata with BlobStandard HTTP metadata (Cache-Control Content-Encoding Content-Type etc)Metadata is ltname valuegt pairs up to 8KB per blob
BLOB CLIENT LIBRARY EXAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudBlobClient blobClient = new CloudBlobClient(
accountBlobEndpoint accountCredentials) Create ContainerCloudBlobContainer cloudContainer = blobClientGetContainerReference(containerName)bool hasCreated = cloudContainerCreateIfNotExist()
Access Blob in the ContainerCloudBlob cloudBlob = cloudContainerGetBlobReference(blobName)
BlobRequestOptions has retry policy timeout etcBlobRequestOptions options = new BlobRequestOptions() Upload the local file to Blob servicecloudBlobUploadFile(uploadFileName options)
Download to local file namecloudBlobDownloadToFile(downloadFileName options)
TWO TYPES OF BLOBS UNDER THE HOOD
Block Blob Targeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedSize limit 200GB per blob
Page BlobTargeted at random readwrite workloadsEach blob consists of an array of pages
Each page range write is committed on PUTSize limit 1TB per blob
BLOCK BLOBStreaming Workload w Random Reads + Committed Writes
Uploading a large blob
10 GB Movie
Windows Azure Storage
Bloc
k Id
1Bl
ock
Id 2
Bloc
k Id
3
Bloc
k Id
N
blobName = ldquoblobwmvrdquoPutBlock(blobName blockId1 block1Bits)PutBlock(blobName blockId2 block2Bits)helliphelliphelliphellipPutBlock(blobName blockIdN blockNBits)PutBlockList(blobName
blockId1 blockId2hellipblockIdN)
blobwmvblobwmv
BenefitUpdate the blob via blocks any way you wantEfficient continuation and retryParallel and out of order upload of blocks
PAGE BLOB ndash RANDOM READWRITECreate MyBlob
Specify Blob Size = 10 GB
Fixed Page Size = 512 bytes
Random Access OperationsPutPage[512 2048)PutPage[0 1024)ClearPage[512 1536)PutPage[20482560)
GetPageRange[0 4096) returns valid data ranges
[0512) [15362560)
GetBlob[1000 2048) returnsAll 0 for first 536 bytesNext 512 bytes are data stored in [15362048)
0
10 GB
10 GB Address Space
512
1024
1536
2048
2560
WINDOWS AZURE CONTENT DELIVERY NETWORK
ScenarioFrequently accessed blobsBlobs accessed from many geographic locations
Windows Azure Content Delivery Network (CDN)Cache and serve your Windows Azure Blobs from our network18 locations globally (US Europe Asia Australia and South America) and growing
BenefitBetter experience for users far from blob serviceProvide high-bandwidth content delivery around the world for popular events
WINDOWS CDN DETAILS
Registering for CDN in Windows Azure Dev Portal Enable storage account to use CDNGet back a domain name and can specify your own custom domain name to use
Windows Azure CDN URL httpltidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom Windows Azure Storage URL httpaccountblobcorewindowsnet
Using Windows Azure CDN for Blob AccessSpecify blobs HTTP Cache-Control policy as TTLMake blobrsquos container(s) publically accessible Reference blobs using CDN domain name
The request is routed to our closest edge locationOn miss or TTL expiration blobs are retrieved from blob service and cached in the CDNAllows the blobs to be served from our CDN caches
SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs but do not want to distribute their secret key
Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs
httpscohowineryblobcorewindowsnetimagespic1jpg
Valid time rangest=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
BLOB TIPSFor high throughput clients
Set ServicePointManagerDefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2UploadDownload multiple files in parallelParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size gt= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KBExpectedThroughputInKBps
Use retry and exponential backoff for timeoutsserver busy
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
WINDOWS AZURE STORAGE ACCOUNT
User creates a globally unique storage account nameCan choose geo-location to host storage account
US ndash ldquoNorth Centralrdquo and ldquoSouth CentralrdquoEurope ndash ldquoNorthrdquo and ldquoWestrdquoAsia ndash ldquoEastrdquo and ldquoSoutheastrdquo
Can co-locate storage account with compute accountReceive a 512 bit secret key when creating account
You get two keys which can be used to rotate
BLOB STORAGE CONCEPTS
BlobContainerAccount
cohowinery
images
PIC01JPG
PIC02JPG
videos VID1AVI
httpltaccountgtblobcorewindowsnetltcontainergtltblobnamegt
BLOB FEATURES AND FUNCTIONS
Programming InterfacesPutBlob
Insert blob and Update blobGetBlob
Get whole blob or a specific rangeDeleteBlobCopyBlobSnapshotBlobLeaseBlob
Associate Metadata with BlobStandard HTTP metadata (Cache-Control Content-Encoding Content-Type etc)Metadata is ltname valuegt pairs up to 8KB per blob
BLOB CLIENT LIBRARY EXAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudBlobClient blobClient = new CloudBlobClient(
accountBlobEndpoint accountCredentials) Create ContainerCloudBlobContainer cloudContainer = blobClientGetContainerReference(containerName)bool hasCreated = cloudContainerCreateIfNotExist()
Access Blob in the ContainerCloudBlob cloudBlob = cloudContainerGetBlobReference(blobName)
BlobRequestOptions has retry policy timeout etcBlobRequestOptions options = new BlobRequestOptions() Upload the local file to Blob servicecloudBlobUploadFile(uploadFileName options)
Download to local file namecloudBlobDownloadToFile(downloadFileName options)
TWO TYPES OF BLOBS UNDER THE HOOD
Block Blob Targeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedSize limit 200GB per blob
Page BlobTargeted at random readwrite workloadsEach blob consists of an array of pages
Each page range write is committed on PUTSize limit 1TB per blob
BLOCK BLOBStreaming Workload w Random Reads + Committed Writes
Uploading a large blob
10 GB Movie
Windows Azure Storage
Bloc
k Id
1Bl
ock
Id 2
Bloc
k Id
3
Bloc
k Id
N
blobName = ldquoblobwmvrdquoPutBlock(blobName blockId1 block1Bits)PutBlock(blobName blockId2 block2Bits)helliphelliphelliphellipPutBlock(blobName blockIdN blockNBits)PutBlockList(blobName
blockId1 blockId2hellipblockIdN)
blobwmvblobwmv
BenefitUpdate the blob via blocks any way you wantEfficient continuation and retryParallel and out of order upload of blocks
PAGE BLOB ndash RANDOM READWRITECreate MyBlob
Specify Blob Size = 10 GB
Fixed Page Size = 512 bytes
Random Access OperationsPutPage[512 2048)PutPage[0 1024)ClearPage[512 1536)PutPage[20482560)
GetPageRange[0 4096) returns valid data ranges
[0512) [15362560)
GetBlob[1000 2048) returnsAll 0 for first 536 bytesNext 512 bytes are data stored in [15362048)
0
10 GB
10 GB Address Space
512
1024
1536
2048
2560
WINDOWS AZURE CONTENT DELIVERY NETWORK
ScenarioFrequently accessed blobsBlobs accessed from many geographic locations
Windows Azure Content Delivery Network (CDN)Cache and serve your Windows Azure Blobs from our network18 locations globally (US Europe Asia Australia and South America) and growing
BenefitBetter experience for users far from blob serviceProvide high-bandwidth content delivery around the world for popular events
WINDOWS CDN DETAILS
Registering for CDN in Windows Azure Dev Portal Enable storage account to use CDNGet back a domain name and can specify your own custom domain name to use
Windows Azure CDN URL httpltidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom Windows Azure Storage URL httpaccountblobcorewindowsnet
Using Windows Azure CDN for Blob AccessSpecify blobs HTTP Cache-Control policy as TTLMake blobrsquos container(s) publically accessible Reference blobs using CDN domain name
The request is routed to our closest edge locationOn miss or TTL expiration blobs are retrieved from blob service and cached in the CDNAllows the blobs to be served from our CDN caches
SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs but do not want to distribute their secret key
Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs
httpscohowineryblobcorewindowsnetimagespic1jpg
Valid time rangest=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
BLOB TIPSFor high throughput clients
Set ServicePointManagerDefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2UploadDownload multiple files in parallelParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size gt= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KBExpectedThroughputInKBps
Use retry and exponential backoff for timeoutsserver busy
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
BLOB STORAGE CONCEPTS
BlobContainerAccount
cohowinery
images
PIC01JPG
PIC02JPG
videos VID1AVI
httpltaccountgtblobcorewindowsnetltcontainergtltblobnamegt
BLOB FEATURES AND FUNCTIONS
Programming InterfacesPutBlob
Insert blob and Update blobGetBlob
Get whole blob or a specific rangeDeleteBlobCopyBlobSnapshotBlobLeaseBlob
Associate Metadata with BlobStandard HTTP metadata (Cache-Control Content-Encoding Content-Type etc)Metadata is ltname valuegt pairs up to 8KB per blob
BLOB CLIENT LIBRARY EXAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudBlobClient blobClient = new CloudBlobClient(
accountBlobEndpoint accountCredentials) Create ContainerCloudBlobContainer cloudContainer = blobClientGetContainerReference(containerName)bool hasCreated = cloudContainerCreateIfNotExist()
Access Blob in the ContainerCloudBlob cloudBlob = cloudContainerGetBlobReference(blobName)
BlobRequestOptions has retry policy timeout etcBlobRequestOptions options = new BlobRequestOptions() Upload the local file to Blob servicecloudBlobUploadFile(uploadFileName options)
Download to local file namecloudBlobDownloadToFile(downloadFileName options)
TWO TYPES OF BLOBS UNDER THE HOOD
Block Blob Targeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedSize limit 200GB per blob
Page BlobTargeted at random readwrite workloadsEach blob consists of an array of pages
Each page range write is committed on PUTSize limit 1TB per blob
BLOCK BLOBStreaming Workload w Random Reads + Committed Writes
Uploading a large blob
10 GB Movie
Windows Azure Storage
Bloc
k Id
1Bl
ock
Id 2
Bloc
k Id
3
Bloc
k Id
N
blobName = ldquoblobwmvrdquoPutBlock(blobName blockId1 block1Bits)PutBlock(blobName blockId2 block2Bits)helliphelliphelliphellipPutBlock(blobName blockIdN blockNBits)PutBlockList(blobName
blockId1 blockId2hellipblockIdN)
blobwmvblobwmv
BenefitUpdate the blob via blocks any way you wantEfficient continuation and retryParallel and out of order upload of blocks
PAGE BLOB ndash RANDOM READWRITECreate MyBlob
Specify Blob Size = 10 GB
Fixed Page Size = 512 bytes
Random Access OperationsPutPage[512 2048)PutPage[0 1024)ClearPage[512 1536)PutPage[20482560)
GetPageRange[0 4096) returns valid data ranges
[0512) [15362560)
GetBlob[1000 2048) returnsAll 0 for first 536 bytesNext 512 bytes are data stored in [15362048)
0
10 GB
10 GB Address Space
512
1024
1536
2048
2560
WINDOWS AZURE CONTENT DELIVERY NETWORK
ScenarioFrequently accessed blobsBlobs accessed from many geographic locations
Windows Azure Content Delivery Network (CDN)Cache and serve your Windows Azure Blobs from our network18 locations globally (US Europe Asia Australia and South America) and growing
BenefitBetter experience for users far from blob serviceProvide high-bandwidth content delivery around the world for popular events
WINDOWS CDN DETAILS
Registering for CDN in Windows Azure Dev Portal Enable storage account to use CDNGet back a domain name and can specify your own custom domain name to use
Windows Azure CDN URL httpltidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom Windows Azure Storage URL httpaccountblobcorewindowsnet
Using Windows Azure CDN for Blob AccessSpecify blobs HTTP Cache-Control policy as TTLMake blobrsquos container(s) publically accessible Reference blobs using CDN domain name
The request is routed to our closest edge locationOn miss or TTL expiration blobs are retrieved from blob service and cached in the CDNAllows the blobs to be served from our CDN caches
SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs but do not want to distribute their secret key
Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs
httpscohowineryblobcorewindowsnetimagespic1jpg
Valid time rangest=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
BLOB TIPSFor high throughput clients
Set ServicePointManagerDefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2UploadDownload multiple files in parallelParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size gt= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KBExpectedThroughputInKBps
Use retry and exponential backoff for timeoutsserver busy
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
BLOB FEATURES AND FUNCTIONS
Programming InterfacesPutBlob
Insert blob and Update blobGetBlob
Get whole blob or a specific rangeDeleteBlobCopyBlobSnapshotBlobLeaseBlob
Associate Metadata with BlobStandard HTTP metadata (Cache-Control Content-Encoding Content-Type etc)Metadata is ltname valuegt pairs up to 8KB per blob
BLOB CLIENT LIBRARY EXAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudBlobClient blobClient = new CloudBlobClient(
accountBlobEndpoint accountCredentials) Create ContainerCloudBlobContainer cloudContainer = blobClientGetContainerReference(containerName)bool hasCreated = cloudContainerCreateIfNotExist()
Access Blob in the ContainerCloudBlob cloudBlob = cloudContainerGetBlobReference(blobName)
BlobRequestOptions has retry policy timeout etcBlobRequestOptions options = new BlobRequestOptions() Upload the local file to Blob servicecloudBlobUploadFile(uploadFileName options)
Download to local file namecloudBlobDownloadToFile(downloadFileName options)
TWO TYPES OF BLOBS UNDER THE HOOD
Block Blob Targeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedSize limit 200GB per blob
Page BlobTargeted at random readwrite workloadsEach blob consists of an array of pages
Each page range write is committed on PUTSize limit 1TB per blob
BLOCK BLOBStreaming Workload w Random Reads + Committed Writes
Uploading a large blob
10 GB Movie
Windows Azure Storage
Bloc
k Id
1Bl
ock
Id 2
Bloc
k Id
3
Bloc
k Id
N
blobName = ldquoblobwmvrdquoPutBlock(blobName blockId1 block1Bits)PutBlock(blobName blockId2 block2Bits)helliphelliphelliphellipPutBlock(blobName blockIdN blockNBits)PutBlockList(blobName
blockId1 blockId2hellipblockIdN)
blobwmvblobwmv
BenefitUpdate the blob via blocks any way you wantEfficient continuation and retryParallel and out of order upload of blocks
PAGE BLOB ndash RANDOM READWRITECreate MyBlob
Specify Blob Size = 10 GB
Fixed Page Size = 512 bytes
Random Access OperationsPutPage[512 2048)PutPage[0 1024)ClearPage[512 1536)PutPage[20482560)
GetPageRange[0 4096) returns valid data ranges
[0512) [15362560)
GetBlob[1000 2048) returnsAll 0 for first 536 bytesNext 512 bytes are data stored in [15362048)
0
10 GB
10 GB Address Space
512
1024
1536
2048
2560
WINDOWS AZURE CONTENT DELIVERY NETWORK
ScenarioFrequently accessed blobsBlobs accessed from many geographic locations
Windows Azure Content Delivery Network (CDN)Cache and serve your Windows Azure Blobs from our network18 locations globally (US Europe Asia Australia and South America) and growing
BenefitBetter experience for users far from blob serviceProvide high-bandwidth content delivery around the world for popular events
WINDOWS CDN DETAILS
Registering for CDN in Windows Azure Dev Portal Enable storage account to use CDNGet back a domain name and can specify your own custom domain name to use
Windows Azure CDN URL httpltidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom Windows Azure Storage URL httpaccountblobcorewindowsnet
Using Windows Azure CDN for Blob AccessSpecify blobs HTTP Cache-Control policy as TTLMake blobrsquos container(s) publically accessible Reference blobs using CDN domain name
The request is routed to our closest edge locationOn miss or TTL expiration blobs are retrieved from blob service and cached in the CDNAllows the blobs to be served from our CDN caches
SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs but do not want to distribute their secret key
Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs
httpscohowineryblobcorewindowsnetimagespic1jpg
Valid time rangest=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
BLOB TIPSFor high throughput clients
Set ServicePointManagerDefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2UploadDownload multiple files in parallelParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size gt= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KBExpectedThroughputInKBps
Use retry and exponential backoff for timeoutsserver busy
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
BLOB CLIENT LIBRARY EXAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudBlobClient blobClient = new CloudBlobClient(
accountBlobEndpoint accountCredentials) Create ContainerCloudBlobContainer cloudContainer = blobClientGetContainerReference(containerName)bool hasCreated = cloudContainerCreateIfNotExist()
Access Blob in the ContainerCloudBlob cloudBlob = cloudContainerGetBlobReference(blobName)
BlobRequestOptions has retry policy timeout etcBlobRequestOptions options = new BlobRequestOptions() Upload the local file to Blob servicecloudBlobUploadFile(uploadFileName options)
Download to local file namecloudBlobDownloadToFile(downloadFileName options)
TWO TYPES OF BLOBS UNDER THE HOOD
Block Blob Targeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedSize limit 200GB per blob
Page BlobTargeted at random readwrite workloadsEach blob consists of an array of pages
Each page range write is committed on PUTSize limit 1TB per blob
BLOCK BLOBStreaming Workload w Random Reads + Committed Writes
Uploading a large blob
10 GB Movie
Windows Azure Storage
Bloc
k Id
1Bl
ock
Id 2
Bloc
k Id
3
Bloc
k Id
N
blobName = ldquoblobwmvrdquoPutBlock(blobName blockId1 block1Bits)PutBlock(blobName blockId2 block2Bits)helliphelliphelliphellipPutBlock(blobName blockIdN blockNBits)PutBlockList(blobName
blockId1 blockId2hellipblockIdN)
blobwmvblobwmv
BenefitUpdate the blob via blocks any way you wantEfficient continuation and retryParallel and out of order upload of blocks
PAGE BLOB ndash RANDOM READWRITECreate MyBlob
Specify Blob Size = 10 GB
Fixed Page Size = 512 bytes
Random Access OperationsPutPage[512 2048)PutPage[0 1024)ClearPage[512 1536)PutPage[20482560)
GetPageRange[0 4096) returns valid data ranges
[0512) [15362560)
GetBlob[1000 2048) returnsAll 0 for first 536 bytesNext 512 bytes are data stored in [15362048)
0
10 GB
10 GB Address Space
512
1024
1536
2048
2560
WINDOWS AZURE CONTENT DELIVERY NETWORK
ScenarioFrequently accessed blobsBlobs accessed from many geographic locations
Windows Azure Content Delivery Network (CDN)Cache and serve your Windows Azure Blobs from our network18 locations globally (US Europe Asia Australia and South America) and growing
BenefitBetter experience for users far from blob serviceProvide high-bandwidth content delivery around the world for popular events
WINDOWS CDN DETAILS
Registering for CDN in Windows Azure Dev Portal Enable storage account to use CDNGet back a domain name and can specify your own custom domain name to use
Windows Azure CDN URL httpltidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom Windows Azure Storage URL httpaccountblobcorewindowsnet
Using Windows Azure CDN for Blob AccessSpecify blobs HTTP Cache-Control policy as TTLMake blobrsquos container(s) publically accessible Reference blobs using CDN domain name
The request is routed to our closest edge locationOn miss or TTL expiration blobs are retrieved from blob service and cached in the CDNAllows the blobs to be served from our CDN caches
SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs but do not want to distribute their secret key
Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs
httpscohowineryblobcorewindowsnetimagespic1jpg
Valid time rangest=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
BLOB TIPSFor high throughput clients
Set ServicePointManagerDefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2UploadDownload multiple files in parallelParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size gt= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KBExpectedThroughputInKBps
Use retry and exponential backoff for timeoutsserver busy
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
TWO TYPES OF BLOBS UNDER THE HOOD
Block Blob Targeted at streaming workloadsEach blob consists of a sequence of blocks
2 Phase commit Blocks are uploaded and then separately committedSize limit 200GB per blob
Page BlobTargeted at random readwrite workloadsEach blob consists of an array of pages
Each page range write is committed on PUTSize limit 1TB per blob
BLOCK BLOBStreaming Workload w Random Reads + Committed Writes
Uploading a large blob
10 GB Movie
Windows Azure Storage
Bloc
k Id
1Bl
ock
Id 2
Bloc
k Id
3
Bloc
k Id
N
blobName = ldquoblobwmvrdquoPutBlock(blobName blockId1 block1Bits)PutBlock(blobName blockId2 block2Bits)helliphelliphelliphellipPutBlock(blobName blockIdN blockNBits)PutBlockList(blobName
blockId1 blockId2hellipblockIdN)
blobwmvblobwmv
BenefitUpdate the blob via blocks any way you wantEfficient continuation and retryParallel and out of order upload of blocks
PAGE BLOB ndash RANDOM READWRITECreate MyBlob
Specify Blob Size = 10 GB
Fixed Page Size = 512 bytes
Random Access OperationsPutPage[512 2048)PutPage[0 1024)ClearPage[512 1536)PutPage[20482560)
GetPageRange[0 4096) returns valid data ranges
[0512) [15362560)
GetBlob[1000 2048) returnsAll 0 for first 536 bytesNext 512 bytes are data stored in [15362048)
0
10 GB
10 GB Address Space
512
1024
1536
2048
2560
WINDOWS AZURE CONTENT DELIVERY NETWORK
ScenarioFrequently accessed blobsBlobs accessed from many geographic locations
Windows Azure Content Delivery Network (CDN)Cache and serve your Windows Azure Blobs from our network18 locations globally (US Europe Asia Australia and South America) and growing
BenefitBetter experience for users far from blob serviceProvide high-bandwidth content delivery around the world for popular events
WINDOWS CDN DETAILS
Registering for CDN in Windows Azure Dev Portal Enable storage account to use CDNGet back a domain name and can specify your own custom domain name to use
Windows Azure CDN URL httpltidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom Windows Azure Storage URL httpaccountblobcorewindowsnet
Using Windows Azure CDN for Blob AccessSpecify blobs HTTP Cache-Control policy as TTLMake blobrsquos container(s) publically accessible Reference blobs using CDN domain name
The request is routed to our closest edge locationOn miss or TTL expiration blobs are retrieved from blob service and cached in the CDNAllows the blobs to be served from our CDN caches
SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs but do not want to distribute their secret key
Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs
httpscohowineryblobcorewindowsnetimagespic1jpg
Valid time rangest=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
BLOB TIPSFor high throughput clients
Set ServicePointManagerDefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2UploadDownload multiple files in parallelParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size gt= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KBExpectedThroughputInKBps
Use retry and exponential backoff for timeoutsserver busy
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
BLOCK BLOBStreaming Workload w Random Reads + Committed Writes
Uploading a large blob
10 GB Movie
Windows Azure Storage
Bloc
k Id
1Bl
ock
Id 2
Bloc
k Id
3
Bloc
k Id
N
blobName = ldquoblobwmvrdquoPutBlock(blobName blockId1 block1Bits)PutBlock(blobName blockId2 block2Bits)helliphelliphelliphellipPutBlock(blobName blockIdN blockNBits)PutBlockList(blobName
blockId1 blockId2hellipblockIdN)
blobwmvblobwmv
BenefitUpdate the blob via blocks any way you wantEfficient continuation and retryParallel and out of order upload of blocks
PAGE BLOB ndash RANDOM READWRITECreate MyBlob
Specify Blob Size = 10 GB
Fixed Page Size = 512 bytes
Random Access OperationsPutPage[512 2048)PutPage[0 1024)ClearPage[512 1536)PutPage[20482560)
GetPageRange[0 4096) returns valid data ranges
[0512) [15362560)
GetBlob[1000 2048) returnsAll 0 for first 536 bytesNext 512 bytes are data stored in [15362048)
0
10 GB
10 GB Address Space
512
1024
1536
2048
2560
WINDOWS AZURE CONTENT DELIVERY NETWORK
ScenarioFrequently accessed blobsBlobs accessed from many geographic locations
Windows Azure Content Delivery Network (CDN)Cache and serve your Windows Azure Blobs from our network18 locations globally (US Europe Asia Australia and South America) and growing
BenefitBetter experience for users far from blob serviceProvide high-bandwidth content delivery around the world for popular events
WINDOWS CDN DETAILS
Registering for CDN in Windows Azure Dev Portal Enable storage account to use CDNGet back a domain name and can specify your own custom domain name to use
Windows Azure CDN URL httpltidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom Windows Azure Storage URL httpaccountblobcorewindowsnet
Using Windows Azure CDN for Blob AccessSpecify blobs HTTP Cache-Control policy as TTLMake blobrsquos container(s) publically accessible Reference blobs using CDN domain name
The request is routed to our closest edge locationOn miss or TTL expiration blobs are retrieved from blob service and cached in the CDNAllows the blobs to be served from our CDN caches
SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs but do not want to distribute their secret key
Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs
httpscohowineryblobcorewindowsnetimagespic1jpg
Valid time rangest=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
BLOB TIPSFor high throughput clients
Set ServicePointManagerDefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2UploadDownload multiple files in parallelParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size gt= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KBExpectedThroughputInKBps
Use retry and exponential backoff for timeoutsserver busy
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
PAGE BLOB ndash RANDOM READWRITECreate MyBlob
Specify Blob Size = 10 GB
Fixed Page Size = 512 bytes
Random Access OperationsPutPage[512 2048)PutPage[0 1024)ClearPage[512 1536)PutPage[20482560)
GetPageRange[0 4096) returns valid data ranges
[0512) [15362560)
GetBlob[1000 2048) returnsAll 0 for first 536 bytesNext 512 bytes are data stored in [15362048)
0
10 GB
10 GB Address Space
512
1024
1536
2048
2560
WINDOWS AZURE CONTENT DELIVERY NETWORK
ScenarioFrequently accessed blobsBlobs accessed from many geographic locations
Windows Azure Content Delivery Network (CDN)Cache and serve your Windows Azure Blobs from our network18 locations globally (US Europe Asia Australia and South America) and growing
BenefitBetter experience for users far from blob serviceProvide high-bandwidth content delivery around the world for popular events
WINDOWS CDN DETAILS
Registering for CDN in Windows Azure Dev Portal Enable storage account to use CDNGet back a domain name and can specify your own custom domain name to use
Windows Azure CDN URL httpltidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom Windows Azure Storage URL httpaccountblobcorewindowsnet
Using Windows Azure CDN for Blob AccessSpecify blobs HTTP Cache-Control policy as TTLMake blobrsquos container(s) publically accessible Reference blobs using CDN domain name
The request is routed to our closest edge locationOn miss or TTL expiration blobs are retrieved from blob service and cached in the CDNAllows the blobs to be served from our CDN caches
SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs but do not want to distribute their secret key
Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs
httpscohowineryblobcorewindowsnetimagespic1jpg
Valid time rangest=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
BLOB TIPSFor high throughput clients
Set ServicePointManagerDefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2UploadDownload multiple files in parallelParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size gt= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KBExpectedThroughputInKBps
Use retry and exponential backoff for timeoutsserver busy
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
WINDOWS AZURE CONTENT DELIVERY NETWORK
ScenarioFrequently accessed blobsBlobs accessed from many geographic locations
Windows Azure Content Delivery Network (CDN)Cache and serve your Windows Azure Blobs from our network18 locations globally (US Europe Asia Australia and South America) and growing
BenefitBetter experience for users far from blob serviceProvide high-bandwidth content delivery around the world for popular events
WINDOWS CDN DETAILS
Registering for CDN in Windows Azure Dev Portal Enable storage account to use CDNGet back a domain name and can specify your own custom domain name to use
Windows Azure CDN URL httpltidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom Windows Azure Storage URL httpaccountblobcorewindowsnet
Using Windows Azure CDN for Blob AccessSpecify blobs HTTP Cache-Control policy as TTLMake blobrsquos container(s) publically accessible Reference blobs using CDN domain name
The request is routed to our closest edge locationOn miss or TTL expiration blobs are retrieved from blob service and cached in the CDNAllows the blobs to be served from our CDN caches
SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs but do not want to distribute their secret key
Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs
httpscohowineryblobcorewindowsnetimagespic1jpg
Valid time rangest=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
BLOB TIPSFor high throughput clients
Set ServicePointManagerDefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2UploadDownload multiple files in parallelParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size gt= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KBExpectedThroughputInKBps
Use retry and exponential backoff for timeoutsserver busy
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
WINDOWS CDN DETAILS
Registering for CDN in Windows Azure Dev Portal Enable storage account to use CDNGet back a domain name and can specify your own custom domain name to use
Windows Azure CDN URL httpltidgtvomsecndnet Custom Domain Name for CDN httpeventscohowinerycom Windows Azure Storage URL httpaccountblobcorewindowsnet
Using Windows Azure CDN for Blob AccessSpecify blobs HTTP Cache-Control policy as TTLMake blobrsquos container(s) publically accessible Reference blobs using CDN domain name
The request is routed to our closest edge locationOn miss or TTL expiration blobs are retrieved from blob service and cached in the CDNAllows the blobs to be served from our CDN caches
SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs but do not want to distribute their secret key
Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs
httpscohowineryblobcorewindowsnetimagespic1jpg
Valid time rangest=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
BLOB TIPSFor high throughput clients
Set ServicePointManagerDefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2UploadDownload multiple files in parallelParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size gt= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KBExpectedThroughputInKBps
Use retry and exponential backoff for timeoutsserver busy
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
SHARED ACCESS SIGNATURES (SIGNED URLS)
Services want to distribute access to blobs but do not want to distribute their secret key
Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs
httpscohowineryblobcorewindowsnetimagespic1jpg
Valid time rangest=Start time (optional)se=End time
Two resource levels of access to grantc=Container | b=Blob
Four types of permissions (or any combination)r=Read | w=Write | d=Delete | l=List
Signed Identifier (optional)Allows time range and permissions to be stored in the blob service for the SASProvides instant revocation of SAS
st=2009-11-07T0849Z ampse=2009-11-07T0949Zampsi=foo
ampsr=c ampsp=rwampsig=3OSeIHP8haK2fle92bBK3BX1DsdMM3d
BLOB TIPSFor high throughput clients
Set ServicePointManagerDefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2UploadDownload multiple files in parallelParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size gt= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KBExpectedThroughputInKBps
Use retry and exponential backoff for timeoutsserver busy
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
BLOB TIPSFor high throughput clients
Set ServicePointManagerDefaultConnectionLimit to allow parallel connections to cloud service
Default value is 2UploadDownload multiple files in parallelParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading
Parallel uploads used when size gt= 32MB
BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW
Timeout = Size in KBExpectedThroughputInKBps
Use retry and exponential backoff for timeoutsserver busy
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
SUMMARY OF WINDOWS AZURE BLOBS
Blob Access PatternsBlock Blobs ndash streaming + commit-based writesPage Blobs ndash random readwrite
Blob OperationsCopy Snapshot and Lease work for both types
Ways of Accessing and Serving Blob ContentContent Delivery Network accessShared Access Signatures (Signed URLs)Custom Domain Names
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
WINDOWS AZURE DRIVEProvides a durable NTFS volume for Windows Azure applications to use
Use existing NTFS APIs to access a network attached durable drive
BenefitsEnables existing applications using NTFS to more easily migrate to the cloudDurability and survival of data on application failover or hardware failure
A Windows Azure Drive is a Page BlobMounts Page Blob over the network as an NTFS driveAll flushed and unbuffered writes to drive are made durable to the Page Blob
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
WINDOWS AZURE DRIVE CAPABILITIES
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)
Drives can be up to 1TB
A Page Blob can only be mounted by one VM at a time for readwriteA VM can dynamically mount up to 16 drives
Remote Access via Page BlobCan upload the VHD to a Page Blob using the blob interface and then mount it as a DriveCan download the Drive through the Page Blob interface
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
VM
HOW WINDOWS AZURE DRIVES WORKS
Windows Azure Blob Service
DemoBlob
Local Cache
OS
Application
Drive is a formatted page blob stored in blob serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushedunbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)Lease
Drive X
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
CLOUD DRIVE CLIENT LIBRARY SAMPLECreate Local Storage resource and initialize the local cache for drivesCloudDriveInitializeCache(localCacheDir cacheSizeInMB)
CloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)
Create a cloud drive (PageBlob)CloudDrive drive = accountCreateCloudDrive(pageBlobUri)driveCreate(1000 sizeInMB )
Mount the network attached drive on the local file systemstring pathOnLocalFS = driveMount(cacheSizeInMB DriveMountOptionsNone)
Use NTFS APIs to ReadWrite files to drivehellip
Snapshot drive while mounted to create backupsUri snapshotUri = driveSnapshot()
Unmount the drivedriveUnmount()
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
WINDOWS AZURE DRIVE SUMMARY
Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud
A Page Blob can only be mounted by one VM at a time for readwrite access
All NTFS non-buffered writes and flushes are made durable to the Page Blob
Can backup and create versions of a drive with Snapshot drive
Can mount a drive snapshot as a read-only VHD with many VMs at the same time
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
WINDOWS AZURE TABLES
Provides Massively Scalable Structured StorageTable can have billions of entities (rows) and TBs of data
Familiar and Easy to use APIWCF (ADONET) Data Services
NET classes and LINQREST ndash with any platform or language
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
TABLE STORAGE CONCEPTS
EntityTableAccount
cohowinery
customers
Name =hellipEmail = hellip
Name =hellipEmail = hellip
winephotos
Photo ID =hellipDate =hellip
Photo ID =hellipDate =hellip
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
ENTITY PROPERTIES
Each entity can have up to 255 properties
Mandatory Properties for every entityPartitionKey amp RowKey
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic Concurrency
No fixed schema for rest of propertiesEach property is stored as a ltname typed valuegt pair
No schema stored for a tableProperties can be the standard NET types
String binary bool DateTime GUID int int64 and double
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
PARTITIONKEY AND PARTITIONS
Every Table has a PartitionKeyIt is the first property (column) of your TableUsed to group entities in the Table into partitions
A Table Partition All entities in a Table with the same partition key valueRowKey provides uniqueness within a partition
PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
PURPOSE OF THE PARTITIONKEY
Entity LocalityEntities in the same partition will be stored together
Efficient querying and cache locality
Entity Group TransactionsAtomically perform multiple InsertUpdateDelete over entities in same partition in a single transaction
Table ScalabilitySystem monitors the usage patterns of partitionsAutomatically load balance partitions
Each partition can be served by a different storage nodeScale to meet the traffic needs of your table
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast amp Furious hellip 2009
Action The Bourne Ultimatum hellip 2007
hellip hellip hellip hellip
Animation Open Season 2 hellip 2009
Animation The Ant Bully hellip 2006
Comedy Office Space hellip 1999
hellip hellip hellip hellip
SciFi X-Men Origins Wolverine hellip 2009
hellip hellip hellip hellip
War Defiance hellip 2008
PARTITIONS AND PARTITION RANGES
Server BTable = Movies
[Comedy - MaxKey)
Server ATable = Movies
[MinKey - Comedy)
Server ATable = Movies
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
TABLE OPERATIONS
TableCreate Query Delete
EntitiesInsertUpdate
Merge ndash Partial updateReplace ndash Update entire entity
DeleteQueryEntity Group Transactions
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
Define the schema as a NET classTABLE SCHEMA
[DataServiceKey(PartitionKey RowKey)] public class Movie Movie Category is the partition key public string PartitionKey get set
Movie Title is the row key public string RowKey get set
public DateTime Timestamp get set
public int ReleaseYear get set public double Rating get set public string Language get set public bool Favorite get set
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
TABLE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudTableClient tableClient = new CloudTableClient( accountTableEndpoint accountCredentials)
Create Movie Tablestring tableName = ldquoMoviesldquotableClientCreateTableIfNotExist(tableName)
TableServiceContext context = tableClientGetDataServiceContext()
Add moviecontextAddObject(tableName new Movie(Action ldquoWhite Water Rapids Survival))contextSaveChangesWithRetries()
Query movievar q = (from movie in contextCreateQueryltMoviegt(tableName) where moviePartitionKey == Action ampamp movieRating gt 40 select movie)AsTableServiceQueryltMoviegt()foreach (Movie movieToUpdate in q) movieToUpdateFavorite = true contextUpdateObject(movieToUpdate)contextSaveChangesWithRetries( )
SaveChangesOptionsBatch
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
TABLE TIPS
Partition your data appropriatelyScaleQueriesEntity Group Transactions
Avoid ldquoAppend onlyrdquo write patterns based on PartitionKey valuesAvoid using monotonically increasing suffix with a constant prefix
Example using only the current timestamp as PartitionKeyIf needed add varying prefix to PartitionKey
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
TABLE TIPSAvoid large table scans when performance is critical
Restructure your schema if requiredConcatenate different keys to form appropriate index
Most OptimalPartitionKey == ldquoSciFirdquo and RowKey == ldquoStar Warsrdquo
Scans Expect continuation tokensPartitionKey == ldquoSciFirdquo and ldquoSphererdquo le RowKey le ldquoStar WarsrdquoldquoActionrdquo le PartitionKey le ldquoThrillerrdquoPartitionKey == ldquoActionrdquo || PartitionKey == ldquoThrillerrdquo - currently scans entire tableldquoCarsrdquo le RowKey le ldquoStar Warsrdquo - scans entire table
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
TABLE TIPS
Default NET HTTP connections is set to 2ServicePointManagerDefaultConnectionLimit = X
Use retries and exponential back-offSaveChangesWithRetries ndash addupdatedeleteAsTableServiceQuery ndash queries
Handle Conflict errors on Inserts and NotFound errors on Delete from retry
Prior try could have completed
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
STORAGE ABSTRACTIONS
Blobs ndash Provide a simple interface for storing named files along with metadata for the file
Drives ndash Provides durable NTFS volumes for Windows Azure applications to use
Tables ndash Provide structured storage A Table is a set of entities which contain a set of properties
Queues ndash Provide reliable storage and delivery of messages for an application
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
QUEUE STORAGE CONCEPTS
MessageQueueAccount
order processing
customer ID order ID httphellip
customer ID order ID httphellip
cohowinery
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
LOOSELY COUPLED WORKFLOW WITH QUEUES
Enables workflow between rolesLoad work in a queue
Producer can forget about message once it is in queueMany workers consume the queue
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
QUEUErsquoS RELIABLE DELIVERY
Guarantee deliveryprocessing of messages (two-step consumption)1 Worker Dequeues message and it is marked as
Invisible for a specified ldquoInvisibility Timerdquo2 Worker Deletes message when finished processing it
If Worker role crashes message becomes visible after the invisibility time for another Worker to process
Azure Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
QUEUE CLIENT LIBRARY SAMPLECloudStorageAccount account = CloudStorageAccountFromConfigurationSetting(CloudStorageAccount)CloudQueueClient queueClient = new CloudQueueClient( accountQueueEndpoint accountCredentials)
Create QueueCloudQueue queue = queueClientGetQueueReference(queueName)queueCreateIfNotExist()
Add MessageCloudQueueMessage message = new CloudQueueMessage(ldquosome content)queueAddMessage(message)
Get Messagemessage = queueGetMessage(TimeSpanFromMinutes(3) Invisibility timeout)
Process Message within the Invisibility Timeout
Delete MessagequeueDeleteMessage(message)
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
QUEUE TIPSMessages can be up to 8KB
Use blob to store large messages and store blob ref in message
A message may be processed more than onceMake message processing idempotent
Work should be repeatable and can be done multiple times
Assume messages put into queue can be processed in any order
For higher throughputBatch multiple work items into a single message or into a blobUse multiple queues
Use DequeueCount to remove poison messagesEnforce threshold on messagersquos dequeue count
Monitor message count to dynamically increasereduce workers
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
Fundamental data abstractions to build your applicationsBlobs ndash Files and large objectsDrives ndash NTFS APIs for migrating applicationsTables ndash Massively scalable structured storageQueues ndash Reliable delivery of messages
Easy to use via the Storage Client Library
Get going at the Windows Azure websitehttpwwwmicrosoftcomwindowsazure
More info on Windows Azure Storage athttpblogsmsdncomwindowsazurestorage
WINDOWS AZURE STORAGE TAKEAWAYS
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
Tribune CompanyJerome SchulistSolutions ArchitectTribune Company
CUSTOMER
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
What is the Tribune Company
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
What has the Tribune Company doneTribune Company (LA Times Chicago Tribune WGN news etc)
Reinventing themselves to stay relevant in a digital age
Storage Needs100 TB of content produced annually25 years of digital historical content137 years of additional history content to digitize
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
What is the value of Windows Azure storageScale elastically and dynamically (out of the disk business)Improve customer experience through reliability and improved performanceQuicker to marketEasy operational management
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
What did they build
Content(Blob Storage)
Image Processor (Queue)
Image Processor(Worker Role)
Content Ingestion Services 1
2
3
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
Track ResourcesWindows Azure Microsoft SQL Azure Windows Azure platform AppFabric wwwwindowsazurecomtechedusLearn More Visit the Windows Azure Boot Camp Room 396
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
Resources
wwwmicrosoftcomteched
Sessions On-Demand amp Community Microsoft Certification amp Training Resources
Resources for IT Professionals Resources for Developers
wwwmicrosoftcomlearning
httpmicrosoftcomtechnet httpmicrosoftcommsdn
Learning
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
Complete an evaluation on CommNet and enter to win
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
Sign up for TechmiddotEd 2011 and save $500 starting June 8 ndash June 31st
httpnorthamericamstechedcomregistration
You can also register at the
North America 2011 kiosk located at registrationJoin us in Atlanta next year
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
copy 2010 Microsoft Corporation All rights reserved Microsoft Windows Windows Vista and other product names are or may be registered trademarks andor trademarks in the US andor other countriesThe information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation Because Microsoft must respond to changing market conditions it should not be interpreted to
be a commitment on the part of Microsoft and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation MICROSOFT MAKES NO WARRANTIES EXPRESS IMPLIED OR STATUTORY AS TO THE INFORMATION IN THIS PRESENTATION
JUNE 7-10 2010 | NEW ORLEANS LA
JUNE 7-10 2010 | NEW ORLEANS LA
