Upload
anton-boyko
View
577
Download
0
Embed Size (px)
Citation preview
Azure Storage
Anton BoykoMicrosoft Azure [email protected]
Agenda
Azure Storage• Overview• Blob Storage• CDN• Drive Storage• Queue Storage• Table Storage• File Storage
Azure Storage Performance• Blob Storage• Drive Storage• Table Storage
OVERVIEW
Storage in the cloud
• Exposed via RESTful web services• Available globally
Storage SDK in many languages
C#/.NET Python Ruby Perl
Node.JS Java PHP Erlang
Common LISP
Objective-C
Storage security
• HTTP or HTTPS• Two independent 512bit symmetric master
keys• Shared access signatures for more granular
access
Storage artifacts
Blobs Drives Queues
Tables Files
BLOBS
Blob storage concept
Block/Page
Blob
Container
Account contoso
image
pic01.jpg
block01 block02
pic02.jpg
video
mov01.avi
page01 page02
https://contoso.blob.core.windows.net/image/pic01.jpg
Blobs main operations
• Get• Put• Delete• Copy
Quiz time
QuestionWhy do I need a “copy” command if it will do pretty much the same as “get” and “put” do together?
Answer
Quiz time
QuestionWhy do I need a “copy” command if it will do pretty much the same as “get” and “put” do together?
Answer“Copy” command is server-side copy. I do not need to download and upload my blob. Also it’s possible to use different storage accounts for source and destination (even in different locations).
Blobs main operations
• Get• Put• Delete• Copy• Snapshot• Lease
Blob types
Streaming workloads
Block Random
access workloads
Page
Block blobs
TheBlob.wmv
10 GB Movie
Bloc
k Id
1Bl
ock
Id 2
Bloc
k Id
3
Bloc
k Id
N
blobName = “TheBlob.wmv”;PutBlock(blobName, blockId1, block1Bits);PutBlock(blobName, blockId2, block2Bits);…………PutBlock(blobName, blockIdN, blockNBits);PutBlockList(blobName,
blockId1,…,blockIdN);
TheBlob.wmv
BenefitEfficient continuation and retryParallel and out of order upload of blocks
THE BLOB
Page blobs
Create MyBlobSpecify Blob Size = 10 GbytesSparse storage - Only charged for pages with data stored in them
Fixed Page Size = 512 bytesRandom Access Operations
PutPage[512, 2048)PutPage[0, 1024)ClearPage[512, 1536)PutPage[2048,2560)
GetPageRange[0, 4096) returns valid data ranges:[0,512) , [1536,2560)
GetBlob[1000, 2048) returnsAll 0 for first 536 bytesNext 512 bytes are data stored in [1536,2048)
0
10 GB
512
1024
1536
2048
2560
10 GB Address Space
Quiz time
QuestionHow can we increase upload speed for page blobs? What is the preferred way to upload 1TB VHD to Azure?
Answer
Quiz time
QuestionHow can we increase upload speed for page blobs? What is the preferred way to upload 1TB VHD to Azure?
AnswerUpload in many threads. Upload only non-zero pages.
CONTENT DELIVERY NETWORK
Content Delivery Network
• High-bandwidth global content delivery• Many origins– Web App– Cloud Service– Storage Account
Quiz time
QuestionCan I use CDN for my VM’s in Azure?
Answer
Quiz time
QuestionCan I use CDN for my VM’s in Azure?
Answer• Yes. All VMs sit behind
Cloud Services.
Content Delivery Network
• High-bandwidth global content delivery• Many origins– Web App– Cloud Service– Storage Account
• New origin– Custom URL
pic1.jpgpic1.jpg
GEThttp://guid01.vo.msecnd.net/images/pic.1jpg
http://sally.blob.core.windows.net/images/pic1.jpg
http://sally.blob.core.windows.net/ http://guid01.vo.msecnd.net/
pic1.jpg
404
TTL Content Delivery Network
Azure Blob Storage
EdgeLocation
EdgeLocation
EdgeLocation
Content Delivery Network
DRIVES
Azure Drives• Durable NTFS volume for Azure Instances
– Use existing NTFS APIs to access a network-attached durable drive– Use System.IO from .NET
• Benefits– Move existing apps that are using NTFS more easily to the cloud– Durability and survival of data on instance recycle – Drives can be up to 1TB
• Azure Drive is NTFS VHD Page Blob– Mounts Page Blob over the network as NTFS drive– Local cache on instance for read operations– All flushed and unbuffered writes to drive are made durable to the Page Blob
TABLES
Table storage concept
Entity
Table
Account contoso
users
FName=BobLName=Smith
FName=IngaPhone=123
groups
Name=Admin
Table details
Table• Create• Query• Delete
Entity• Insert• Update
– Merge– Replace
• Upsert• Query• Delete
Entity properties• Entity can have up to 255 properties
– Up to 1MB per entity
• Mandatory Properties for every entity– PartitionKey & RowKey (only indexed properties)
• Uniquely identifies an entity• Defines the sort order
– Timestamp • Optimistic Concurrency• Exposed as an HTTP Etag
• No fixed schema for other properties– Each property is stored as a <name, typed value> pair– No schema stored for a table– Properties can be the standard .NET types
String, binary, bool, DateTime, GUID, int and double
Purpose of Partition Key• Entity Locality
– Entities in the same partition will be stored together– Efficient querying and cache locality– Endeavour to include partition key in all queries
• Entity Group Transactions– Atomic multiple Insert/Update/Delete in the same partition in a single transaction
• Table Scalability– Target throughput – 2 000 tps/partition, 20 000 tps/account– Azure monitors usage patterns of partitions– Automatically load balance partitions– Each partition can be served by a different storage node– Scale to meet traffic needs of your table
Poker scoring table
By players• PK = PlayerID• RK = GameID• Score = 42
By games• PK = GameID• RK = PlayerID• Score = 42
there is no silver bullet
Blog tables
Posts
• PK = Date• RK = Time• Authors• Tags• Text
PostsByAuthors
• PK = UserID• RK = Post PK +
RK
PostsByTags
• PK = Tag• RK = Post PK +
RK
IT’S SHOW TIME
DRIVES
Single drive
4 * 1/4 drives
BLOBS
Upload single-threadMbytes uploaded: 142.49238300323486total time in ms: 7377avg ms per Mbyte: 51.771188357714024
Mbytes uploaded: 1424.9273290634155total time in ms: 70467avg ms per Mbyte: 49.45304828023543
Upload multi-threadtotal files uploaded: 365total Mbytes uploaded: 142.4734115600586total time in ms: 4576avg ms per Mbyte: 32.11827350727137
total files uploaded: 365total Mbytes uploaded: 1424.9266424179077total time in ms: 36323avg ms per Mbyte: 25.491136819762723
Copy blobMbytes uploaded: 127000total time in ms: 177avg ms per Mbyte: 0.0013937007874015748
Mbytes uploaded: 127000total time in ms: 1836avg ms per Mbyte: 0.014456692913385827
TABLES
Insert single entityrecords inserted: 500total time in ms: 2920avg ms per record: 5.84
Insert batch of 50records inserted: 500total time in ms: 470avg ms per record: 0.94
Insert batch of 100 (9100 times)records inserted: 910000total time in ms: 1072417 (~18 min)avg ms per record: 1.1784802197802198
Select all single-threadrecords selected: 1500queries count: 2total time in ms: 274avg ms per record: 0.18266666666666667
Select single partitionrecords selected: 5000queries count: 5total time in ms: 815avg ms per record: 0.163
Select all single-threadrecords selected: 1142100queries count: 1143total time in ms: 335824 (~6 min)avg ms per record: 0.29404080203134575
Select all multi-threadrecords selected: 1142100queries count partition: 306queries count data: 1259total time in ms: 48134 (~48 sec)avg ms per record: 0.04214517117590403
total http queries: +37%total time in ms: -76%avg ms per record: -76%
Select by custom propertyrecords selected: 284927queries count: 285total time in ms: 74065avg ms per record: 0.2599437750722115
QuestionsAnton BoykoMicrosoft Azure [email protected]
facebook.com/boyko.antyoutube.com/user/boykoant
Google: boykoant