Upload
lorna
View
79
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Windows Azure Storage. Anton Boyko .NET developer. Agenda. Windows Azure Storage Blob Storage Drives Tables. Windows Azure Storage. Storage in the Cloud Scalable, durable, and available Anywhere at anytime access Only pay for what the service uses Exposed via RESTful Web Services - PowerPoint PPT Presentation
Citation preview
Windows Azure StorageAnton Boyko.NET developer
Agenda
Windows Azure StorageBlob StorageDrivesTables
Windows Azure StorageStorage in the CloudScalable, durable, and availableAnywhere at anytime accessOnly pay for what the service uses
Exposed via RESTful Web ServicesUse from Windows Azure ComputeUse from anywhere on the internet
Windows Azure Storage AccountUser specified globally unique account name
North Central US Northern Europe
Western Europe East Asia
South East Asia
US Europe Asia
Can choose geo-location to host storage account:
South Central US
West US East US
Windows Azure Storage AccountCan CDN Enable AccountBlobs delivered via 24 global CDN nodes
Can co-locate storage account with compute accountExplicitly or using affinity groups
Accounts have two independent 512 bit shared secret keys
100 TBs per account
New FeaturesGeo-ReplicationStorage AnalyticsLogs: Provide trace of executed requests for your storage accountsMetrics: Provide summary of key capacity and request statistics for Blobs, Tables, and Queues
Storage in the Development FabricProvides a local “Mock” storageEmulates storage in cloudAllows offline developmentRequires SQL Express 2005/2008 or above
http://msdn.microsoft.com/en-us/gg433135
The Storage Client APIIn this presentation we’ll cover the underlying
RESTful APICan call these from any HTTP cliente.g. Flash, Silverlight, etc…
Client API from SDK Microsoft.WindowsAzure.StorageClientProvides a strongly typed wrapper around REST services
Storage SecurityWindows Azure Storage provides simple security for calls to storage serviceHTTPS endpointDigitally sign requests for privileged operations
Two 512bit symmetric keys per storage accountCan be regenerated independently
More granular security via Shared Access Signatures
Windows Azure Storage Abstractions
TablesStructured storage. A table is a set of entities; an entity is
a set of properties.
QueuesReliable storage and delivery of messages for an application.
BlobsSimple named files along with metadata for the file.
DrivesDurable NTFS volumes for Windows Azure applications to use. Based on Blobs.
Blob Storage
Blob Storage Concepts
BlobContainerAccount
http://<account>.blob.core.windows.net/<container>/<blobname>
Pages/ Blocks
contoso
PIC01.JPG
Block/Page
Block/Page
PIC02.JPGimages
VID1.AVIvideos
Blob Details
Main Web Service
Operations
PutBlobGetBlobDeleteBlobCopyBlobSnapshotBlob LeaseBlob
Blob Details
Associate Metadata with Blob
Standard HTTP metadata/headers (Cache-Control, Content-Encoding, Content-Type, etc)
Metadata is <name, value> pairs, up to 8KB per blob
Either as part of PutBlob or independently
Blob Details
Blob always accessed by
name
Can include ‘/‘ or other delimeter in name e.g. /<container>/myblobs/blob.jpg
Blob ContainersMultiple Containers per AccountSpecial $root container
Blob ContainerA container holds a set of blobsSet access policies at the container level Associate Metadata with ContainerList the blobs in a containerIncluding Blob Metadata and MD5 NO search/query. i.e. no WHERE MetadataValue = ?
Blobs ThroughputEffectively in Partition of 1Target of 60MB/s per Blob
GET http://.../products?comp=list&prefix=Tents&delimiter=/
<Blob>Tents/PalaceTent.wmv</Blob><Blob>Tents/ShedTent.wmv</Blob>
Enumerating Blobs
GET Blob operation takes parametersPrefixDelimiterInclude= (snapshots, metadata etc…)
http://adventureworks.blob.core.windows.net/ Products/Bikes/SuperDuperCycle.jpg Products/Bikes/FastBike.jpg Products/Canoes/Whitewater.jpg Products/Canoes/Flatwater.jpg Products/Canoes/Hybrid.jpg Products/Tents/PalaceTent.jpg Products/Tents/ShedTent.jpg
http://.../products?comp=list&prefix=Canoes&maxresults=2
&marker=MarkerValue
<Blob>Canoes/Hybrid.jpg</Blob>
Pagination
Large lists of Blobs can
be paginatedEither set maxresults or;Exceed default value for maxresults (5000)
http://.../products?comp=list&prefix=Canoes&maxresults=2
<Blob>Canoes/Whitewater.jpg</Blob><Blob>Canoes/Flatwater.jpg</Blob><NextMarker>MarkerValue</NextMarker>
Two Types of Blobs Under the Hood
Block BlobTargeted at streaming workloads
Each blob consists of a sequence of blocksEach block is identified by a Block ID
Size limit 200GB per blob
Optimistic Concurrency via Etags
Page BlobTargeted at random read/write workloads
Each blob consists of an array of pages Each page is identified by its offset from the start of the blob
Size limit 1TB per blob
Optimistic or Pessimistic (locking) concurrency via leases
TheBlob.wmv
Uploading a Block BlobUploading a large blob
10 GB Movie
Bloc
k Id
1Bl
ock
Id 2
Bloc
k Id
3
Bloc
k Id
N
blobName = “TheBlob.wmv”;PutBlock(blobName, blockId1, block1Bits);PutBlock(blobName, blockId2, block2Bits);…………PutBlock(blobName, blockIdN, blockNBits);PutBlockList(blobName,
blockId1,…,blockIdN);
TheBlob.wmv
BenefitEfficient continuation and retryParallel and out of order upload of blocks
THE BLOB
Windows AzureStorage
Page Blob – Random Read/WriteCreate MyBlob
Specify Blob Size = 10 GbytesSparse storage - Only charged for pages with data stored in them
Fixed Page Size = 512 bytesRandom Access Operations
PutPage[512, 2048)PutPage[0, 1024)ClearPage[512, 1536)PutPage[2048,2560)
GetPageRange[0, 4096) returns valid data ranges:
[0,512) , [1536,2560)GetBlob[1000, 2048) returns
All 0 for first 536 bytesNext 512 bytes are data stored in [1536,2048)
0
10 GB
5121024153620482560
10 GB Address Space
Shared Access SignaturesFine grain access rights to blobs and containersSign URL with storage key – permit elevated rightsRevocationUse short time periods and re-issueUse container level policy that can be deleted
Two broad approachesAd-hocPolicy based
Content Delivery Network (CDN)High-bandwidth global blob content delivery24 locations globally (US, Europe, Asia, Australia and South America), and growingSame experience for users no matter how far they are from the geo-location where the storage account is hosted
Blob service URL vs. CDN URL:Windows Azure Blob URL: http://images.blob.core.windows.net/Windows Azure CDN URL: http://<id>.vo.msecnd.net/ Custom Domain Name for CDN: http://cdn.contoso.com/
pic1.jpg
Windows Azure CDN
To Enable CDN:Register for CDN via Dev PortalSet container images to public
pic1.jpg
GEThttp://guid01.vo.msecnd.net/images/pic.1jpg
http://sally.blob.core.windows.net/images/pic1.jpg
http://sally.blob.core.windows.net/ http://guid01.vo.msecnd.net/
pic1.jpg
404
TTL Content Delivery Network
Windows Azure Blob Service
EdgeLocation
EdgeLocation
EdgeLocation
Drives
Windows Azure DrivesDurable NTFS volume for Windows Azure InstancesUse existing NTFS APIs to access a network attached durable driveUse System.IO from .NET
BenefitsMove existing apps using NTFS more easily to the cloudDurability and survival of data on instance recycle Drives can be up to 1TB
A Windows Azure Drive is an NTFS VHD Page BlobMounts Page Blob over the network as an NTFS driveLocal cache on instance for read operationsAll flushed and unbuffered writes to drive are made durable to the Page Blob
Drive DetailsOperations performed via Drive API not REST CallsOperations on DrivesCreateDriveCreates a new NTFS formatted VHD in Blob storage
MountDrive/UnmountDriveMounts a drive into Instance at new drive letterUnmounts a drive freeing drive letter
Get Mounted DrivesList mounted drives; underlying blob and drive letter
Snapshot DriveCreate snapshot copy of the drive
VM
How Windows Azure Drives Works Drive is a formatted page blob stored in blob
serviceMount obtains a blob lease Mount specifies amount of local storage for cacheNTFS flushed/unbuffered writes commit to blob store before returning to appNTFS reads can be served from local cache or from blob store (cache miss)
DemoBlob
OS
Application
Drive X:
Windows Azure Blob Service
Local Cache
Failover with DrivesMust issue NTFS Flush command
to persist dataUse System.IO.Stream.Flush()
Read/Write Drives protected with leases1 Minute lease expiryMaintained by Windows Azure OS DriverUnmount on RoleEntryPoint.OnStop
On failureLease will timeout after 1 minuteRe-mount drive on new instance
Tables
Table Storage ConceptsEntityTableAccount
contoso
Name =…Email = …
Name =…EMailAdd=
customers
Photo ID =…Date =…
photos
Photo ID =…Date =…
Table Details
InsertUpdate Merge – Partial updateReplace – Update entire entityUpsertDeleteQueryEntity Group TransactionsMultiple CUD Operations in a single atomic transaction
Create, Query, DeleteTables can have metadata
Not an RDBMS! Table
Entities
Entity PropertiesEntity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic ConcurrencyExposed as an HTTP Etag
No fixed schema for other propertiesEach property is stored as a <name, typed value> pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double
No Fixed Schema
FIRST LAST BIRTHDATE
Wade Wegner 2/2/1981
Nathan Totten 3/15/1965
Nick Harris May 1, 1976
FAV SPORT
Canoeing
Querying
FIRST LAST BIRTHDATE
Wade Wegner 2/2/1981
Nathan Totten 3/15/1965
Nick Harris May 1, 1976
?$filter=Last eq ‘Wegner’
Purpose of the PartitionKeyEntity LocalityEntities in the same partition will be stored togetherEfficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple Insert/Update/Delete in same partition in a single transaction
Table ScalabilityTarget throughput – 500 tps/partition, several thousand tps/accountWindows Azure monitors the usage patterns of partitionsAutomatically load balance partitionsEach partition can be served by a different storage nodeScale to meet the traffic needs of your table
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE) TIMESTAMP MODELYEAR
Bikes Super Duper Cycle … 2009
Bikes Quick Cycle 200 Deluxe … 2007
… … … …Canoes Whitewater … 2009Canoes Flatwater … 2006
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE) TIMESTAMP MODELYEAR
Rafts 14ft Super Tourer … 1999
… … … …
Skis Fabrikam Back Trackers … 2009
… … … …Tents Super Palace … 2008
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE) TIMESTAMP MODELYEAR
Bikes Super Duper Cycle … 2009
Bikes Quick Cycle 200 Deluxe … 2007
… … … …Canoes Whitewater … 2009Canoes Flatwater … 2006Rafts 14ft Super Tourer … 1999… … … …
Skis Fabrikam Back Trackers … 2009
… … … …Tents Super Palace … 2008
Partitions and Partition Ranges
Server ATable = Products
Server BTable = Products
[Canoes - MaxKey)
Server ATable = Products
[MinKey - Canoes)