View
223
Download
0
Category
Tags:
Preview:
Citation preview
Windows Azure Storage
NameTitleMicrosoft Corporation
Agenda
Windows Azure Storage
Blob Storage
Drives
Tables
Queues
Windows Azure Storage
Storage in the CloudScalable, durable, and availableAnywhere at anytime accessOnly pay for what the service uses
Exposed via RESTful Web ServicesUse from Windows Azure ComputeUse from anywhere on the internet
Windows Azure Storage AccountUser specified globally unique account name
North Central US
South Central US
Northern Europe
Western Europe East Asia
South East Asia
US Europe Asia
Can choose geo-location to host storage account:
Windows Azure Storage AccountCan CDN Enable AccountBlobs delivered via 24 global CDN nodes
Can co-locate storage account with compute accountExplicitly or using affinity groups
Accounts have two independent 512 bit shared secret keys
100 TBs per account
New Features
Geo-ReplicationStorage AnalyticsLogs: Provide trace of executed requests for your storage accountsMetrics: Provide summary of key capacity and request statistics for Blobs, Tables, and Queues
Improved HTTP headers for Blobs
Storage in the Development FabricProvides a local “Mock” storageEmulates storage in cloudAllows offline developmentRequires SQL Express 2005/2008 or above
http://msdn.microsoft.com/en-us/gg433135
The Storage Client API
In this presentation we’ll cover the underlying
RESTful APICan call these from any HTTP cliente.g. Flash, Silverlight, etc…
Client API from SDK Microsoft.WindowsAzure.StorageClientProvides a strongly typed wrapper around REST services
Storage Libraries in Many Languages
Storage Security
Windows Azure Storage provides simple security for calls to storage serviceHTTPS endpointDigitally sign requests for privileged operations
Two 512bit symmetric keys per storage accountCan be regenerated independently
More granular security via Shared Access Signatures
Windows Azure Storage Abstractions
TablesStructured storage. A table is a set of entities; an entity is
a set of properties.
QueuesReliable storage and delivery of messages for an application.
BlobsSimple named files along with metadata for the file.
DrivesDurable NTFS volumes for Windows Azure applications to use. Based on Blobs.
Blob Storage
Blob Storage Concepts
BlobContainerAccount
http://<account>.blob.core.windows.net/<container>/<blobname>
Pages/ Blocks
contoso
PIC01.JPG
Block/Page
Block/Page
PIC02.JPG
images
VID1.AVIvideos
Blob Details
Main Web Service
Operations
PutBlobGetBlobDeleteBlobCopyBlobSnapshotBlob LeaseBlob
Blob Details
Associate Metadata with Blob
Standard HTTP metadata/headers (Cache-Control, Content-Encoding, Content-Type, etc)
Metadata is <name, value> pairs, up to 8KB per blob
Either as part of PutBlob or independently
Blob Details
Blob always accessed by
name
Can include ‘/‘ or other delimeter in name e.g. /<container>/myblobs/blob.jpg
Blob ContainersMultiple Containers per AccountSpecial $root container
Blob ContainerA container holds a set of blobsSet access policies at the container level Associate Metadata with ContainerList the blobs in a containerIncluding Blob Metadata and MD5 NO search/query. i.e. no WHERE MetadataValue = ?
Blobs ThroughputEffectively in Partition of 1Target of 60MB/s per Blob
GET http://.../products?comp=list&prefix=Tents&delimiter=/
<Blob>Tents/PalaceTent.wmv</Blob><Blob>Tents/ShedTent.wmv</Blob>
Enumerating Blobs
GET Blob operation takes parametersPrefixDelimiterInclude= (snapshots, metadata etc…)
http://adventureworks.blob.core.windows.net/ Products/Bikes/SuperDuperCycle.jpg Products/Bikes/FastBike.jpg Products/Canoes/Whitewater.jpg Products/Canoes/Flatwater.jpg Products/Canoes/Hybrid.jpg Products/Tents/PalaceTent.jpg Products/Tents/ShedTent.jpg
http://.../products?comp=list&prefix=Canoes&maxresults=2
&marker=MarkerValue
<Blob>Canoes/Hybrid.jpg</Blob>
Pagination
Large lists of Blobs can
be paginatedEither set maxresults or;Exceed default value for maxresults (5000)
http://.../products?comp=list&prefix=Canoes&maxresults=2
<Blob>Canoes/Whitewater.jpg</Blob><Blob>Canoes/Flatwater.jpg</Blob><NextMarker>MarkerValue</NextMarker>
Tour of the Blob Service
demo
Two Types of Blobs Under the Hood
Block BlobTargeted at streaming workloads
Each blob consists of a sequence of blocksEach block is identified by a Block ID
Size limit 200GB per blob
Optimistic Concurrency via Etags
Page BlobTargeted at random read/write workloads
Each blob consists of an array of pages Each page is identified by its offset from the start of the blob
Size limit 1TB per blob
Optimistic or Pessimistic (locking) concurrency via leases
TheBlob.wmv
Uploading a Block Blob
Uploading a large blob
10 GB Movie
Blo
ck I
d 1
Blo
ck I
d 2
Blo
ck I
d 3
Blo
ck I
d N
blobName = “TheBlob.wmv”;PutBlock(blobName, blockId1, block1Bits);PutBlock(blobName, blockId2, block2Bits);…………PutBlock(blobName, blockIdN, blockNBits);PutBlockList(blobName,
blockId1,…,blockIdN);
TheBlob.wmv
BenefitEfficient continuation and retryParallel and out of order upload of blocks
THE BLOB
Windows AzureStorage
Page Blob – Random Read/Write
Create MyBlobSpecify Blob Size = 10 GbytesSparse storage - Only charged for pages with data stored in them
Fixed Page Size = 512 bytesRandom Access Operations
PutPage[512, 2048)PutPage[0, 1024)ClearPage[512, 1536)PutPage[2048,2560)
GetPageRange[0, 4096) returns valid data ranges:
[0,512) , [1536,2560)GetBlob[1000, 2048) returns
All 0 for first 536 bytesNext 512 bytes are data stored in [1536,2048)
0
10 GB
512
1024
1536
2048
2560
10 G
B A
dd
ress S
pace
Shared Access Signatures
Fine grain access rights to blobs and containersSign URL with storage key – permit elevated rightsRevocationUse short time periods and re-issueUse container level policy that can be deleted
Two broad approachesAd-hocPolicy based
Ad Hoc SignaturesCreate Short Dated Shared Access SignatureSignedresource Blob or ContainerAccessPolicy Start, Expiry and PermissionsSignature HMAC-SHA256 of above fields
Use caseSingle use URLsE.g. Provide URL to Silverlight client to upload to container
http://...blob.../pics/image.jpg?sr=c&st=2009-02-09T08:20Z&se=2009-02-10T08:30Z&sp=w
&sig= dD80ihBh5jfNpymO5Hg1IdiJIEvHcJpCMiCMnN%2fRnbI%3d
Create Container Level Policy Specify StartTime, ExpiryTime, Permissions
Create Shared Access Signature URLSignedresource Blob or ContainerSignedidentifier Optional pointer to container policySignature HMAC-SHA256 of above fields
Use caseProviding revocable permissions to certain users/groupsTo revoke: Delete or update container policy
http://...blob.../pics/image.jpg?sr=c&si=MyUploadPolicyForUserID12345
&sig=dD80ihBh5jfNpymO5Hg1IdiJIEvHcJpCMiCMnN%2fRnbI%3d
Policy Based Signatures
Content Delivery Network (CDN)
High-bandwidth global blob content delivery24 locations globally (US, Europe, Asia, Australia and South America), and growing
Same experience for users no matter how far they are from the geo-location where the storage account is hosted
Blob service URL vs. CDN URL:Windows Azure Blob URL: http://images.blob.core.windows.net/
Windows Azure CDN URL: http://<id>.vo.msecnd.net/
Custom Domain Name for CDN: http://cdn.contoso.com/
pic1.jpg
Windows Azure CDN
To Enable CDN:Register for CDN via Dev PortalSet container images to public
pic1.jpg
GEThttp://guid01.vo.msecnd.net/images/pic.1jpg
http://sally.blob.core.windows.net/images/pic1.jpg
http://sally.blob.core.windows.net/ http://guid01.vo.msecnd.net/
pic1.jpg
404
TTL Content Delivery Network
Windows Azure Blob Service
EdgeLocation
EdgeLocation
EdgeLocation
Drives
Windows Azure Drives
Durable NTFS volume for Windows Azure InstancesUse existing NTFS APIs to access a network attached durable driveUse System.IO from .NET
BenefitsMove existing apps using NTFS more easily to the cloudDurability and survival of data on instance recycle
A Windows Azure Drive is an NTFS VHD Page BlobMounts Page Blob over the network as an NTFS driveLocal cache on instance for read operationsAll flushed and unbuffered writes to drive are made durable to the Page Blob
Windows Azure Drive Capabilities
A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)Drives can be up to 1TB
A Page Blob can be mounted:On one instance at a time for read/writeUsing read-only snapshots to multiple instances at once
Windows Azure Drive Capabilities
An instance can dynamically mount up
to 16 drivesRemote Access via standard BlobUICan’t remotely mount driveCan upload the VHD to a Page Blob using the blob interface, and then mount it as a DriveCan download the VHD to a local file and mount locally
Drive Details
Operations performed via Drive API not REST CallsOperations on DrivesCreateDriveCreates a new NTFS formatted VHD in Blob storage
MountDrive/UnmountDriveMounts a drive into Instance at new drive letter
Unmounts a drive freeing drive letter
Get Mounted DrivesList mounted drives; underlying blob and drive letter
Snapshot DriveCreate snapshot copy of the drive
VM
How Windows Azure Drives Works Drive is a formatted page blob stored in blob
service
Mount obtains a blob lease
Mount specifies amount of local storage for cache
NTFS flushed/unbuffered writes commit to blob store before returning to app
NTFS reads can be served from local cache or from blob store (cache miss)
DemoBlob
OS
Application
Drive X:
Windows Azure Blob Service
Local Cache
Cloud Drive Client Library Sample
CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount");
//Initialize the local cache for drives mounted by this role instanceCloudDrive.InitializeCache(localCacheDir, cacheSizeInMB);
//Create a cloud drive (PageBlob)CloudDrive drive = account.CreateCloudDrive(pageBlobUri);drive.Create(1000 /* sizeInMB */);
//Mount the network attached drive on the local file systemstring pathOnLocalFS = drive.Mount(cacheSizeInMB, DriveMountOptions.None);
//Use NTFS APIs to Read/Write files to drive
…
//Snapshot drive while mounted to create backupsUri snapshotUri = drive.Snapshot();
//Unmount the drivedrive.Unmount();
Failover with Drives
Must issue NTFS Flush command
to persist dataUse System.IO.Stream.Flush()
Read/Write Drives protected with leases1 Minute lease expiryMaintained by Windows Azure OS DriverUnmount on RoleEntryPoint.OnStop
On failureLease will timeout after 1 minuteRe-mount drive on new instance
Tables
Table Storage Concepts
EntityTableAccount
contoso
Name =…Email = …
Name =…EMailAdd=
customers
Photo ID =…Date =…
photos
Photo ID =…Date =…
Table Details
InsertUpdate Merge – Partial update
Replace – Update entire entity
UpsertDeleteQueryEntity Group TransactionsMultiple CUD Operations in a single atomic transaction
Create, Query, DeleteTables can have metadata
Not an RDBMS! Table
Entities
Entity PropertiesEntity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)Uniquely identifies an entityDefines the sort order
Timestamp Optimistic ConcurrencyExposed as an HTTP Etag
No fixed schema for other propertiesEach property is stored as a <name, typed value> pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double
No Fixed Schema
FIRST LAST BIRTHDATE
Wade Wegner 2/2/1981
Nathan Totten 3/15/1965
Nick Harris May 1, 1976
FAV SPORT
Canoeing
Querying
FIRST LAST BIRTHDATE
Wade Wegner 2/2/1981
Nathan Totten 3/15/1965
Nick Harris May 1, 1976
?$filter=Last eq ‘Wegner’
Purpose of the PartitionKeyEntity LocalityEntities in the same partition will be stored togetherEfficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple Insert/Update/Delete in same partition in a single transaction
Table ScalabilityTarget throughput – 500 tps/partition, several thousand tps/accountWindows Azure monitors the usage patterns of partitionsAutomatically load balance partitionsEach partition can be served by a different storage nodeScale to meet the traffic needs of your table
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE)
TIMESTAMP MODELYEAR
Bikes Super Duper Cycle … 2009
BikesQuick Cycle 200 Deluxe
… 2007
… … … …
Canoes Whitewater … 2009
Canoes Flatwater … 2006
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE)
TIMESTAMP MODELYEAR
Rafts 14ft Super Tourer … 1999
… … … …
SkisFabrikam Back Trackers
… 2009
… … … …
Tents Super Palace … 2008
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE)
TIMESTAMP MODELYEAR
Bikes Super Duper Cycle … 2009
BikesQuick Cycle 200 Deluxe
… 2007
… … … …
Canoes Whitewater … 2009
Canoes Flatwater … 2006
Rafts 14ft Super Tourer … 1999
… … … …
SkisFabrikam Back Trackers
… 2009
… … … …
Tents Super Palace … 2008
Partitions and Partition Ranges
Server ATable = Products
Server BTable = Products
[Canoes - MaxKey)
Server ATable = Products
[MinKey - Canoes)
Queues
Queue Storage Concepts
MessageTableAccount
customer ID order ID http://…
adventureworks
order processing
customer ID order ID http://…
Loosely Coupled Workflow with QueuesEnables workflow between rolesLoad work in a queueProducer can forget about message once it is in queueMany workers consume the queueFor extreme throughput (>500 tps) Use multiple queuesRead messages in batchesMultiple work items per message
Queue
Input Queue (Work Items)
Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
Queue Details
Simple asynchronous dispatch queueNo limit to queue length subject to storage limit64kb per messageListQueues - List queues in account
Queue operations CreateQueue DeleteQueueGet/Set MetadataClear Messages
Queue Details
Message operationsPutMessage– Reads message and hides for time periodGetMessages – Reads one or more messages and hides themPeekMessages – Reads one or more messages w/o hiding themDeleteMessage – Permanently deletes messages from queueUpdateMessage – Clients renew the lease and contents
Queue’s Reliable Delivery
Guarantee delivery/processing of messages (two-step consumption)Worker queues message and it is marked as Invisible for a specified “Invisibility Time”Worker Deletes message when finished processingIf Worker role crashes, message becomes visible for another Worker to process
Windows Azure Storage SummaryFundamental data abstractions to build your applicationsBlobs: Files and large objectsDrives: NTFS APIs for migrating applicationsTables: Massively scalable structured storageQueues: Reliable delivery of messages
Easy to use via the Storage Client Library
Hands on Labs
© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to
be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Recommended