Embed Size (px)
Citation preview
K.Mohamed FaizalLead Consultant @ NCS (P) [email protected]://faizal-comeacross.blogspot.com
AzureUG.SG 26th Jan 2011.
Exploring Windows Azure Cloud Storage
About Me
14
Agenda
Cloud Storage Account OverviewTablesBlobsQueuesStorage Viewer / Explorer Tools
Cloud Storage
The types of Azure Storage• Blobs – A file system• Tables – Hierarchical Data Tables• Queues – Asynchronous Messaging• Drives – NTFS file systems (Blob)Each item in storage is stored in
triplicate.
Blobs QueuesTables
Make a guess!
What is the maximum size of data, a storage account can hold?a) 1 GBb) 100 GBc) 1 TBd) 100 TB
http://blogs.msdn.com/b/windowsazurestorage/archive/2010/05/10/windows-azure-storage-abstractions-and-their-scalability-targets.aspx
Answer: d) 100 TB
Windows Azure and SQL Azure
Azure Storage SQL Azure
Vision Highly scalable, highly available store in the Cloud
Scalable, highly available relational store in the Cloud
Access Uses WCF Data Services - REST
SqlClient + TSQL
Relational? No Yes – but with some limitations
Analogy File System RDBMS – as it is
Maximum Amount of data in a single “database”
100TB 50GB (up to Oct 2010)
Price per GB per month
$ 0.15 $ 9.99
Creating a Storage Account
With client libraryAutomatically referenced in a cloud projectSupplies a nice .NET API on top of the REST API
You will need the following:Account Name - movieconversionAccount Shared Key – NOcqFlqUwJPLlm…Endpoint - blob.core.windows.net
-or-UseDevelopmentStorage=true
?Table Storage
Windows Azure TablesProvides Structured Storage
Massively Scalable TablesBillions of entities (rows) and TBs of dataCan use thousands of servers as traffic grows
Highly Available & DurableData is replicated several times
Familiar and Easy to use APIADO.NET Data Services – .NET 3.5 SP1
.NET classes and LINQREST – with any platform or language10
TablesIs not relationalCan Not-
Create foreign key relationships between tables.Perform server side joins between tables.Create custom indexes on the tables.No server side Count(), for example.
ADO.NET Data ServiceCreates and consumes data via REST-based API'sThe Table Service REST API implements the WCF Data Services API.
Table Structure
Account: MovieData
Star WarsStar TrekFan Boys
Table Name: Movies
Brian H. PrinceJason ArgonautBill Gates
Table Name: Customers
Account
Table
Entity Tables store entities.
Table Storage ConceptsEntitiesTablesAccounts
moviesonline
Users
Movies
Email =…Name = …
Email =…Name = …
Genre =…Title = …
Genre =…Title = …
13
Table Data ModelTable
A storage account can create many tablesTable name is scoped by accountSet of entities (i.e. rows)
EntitySet of properties (columns)Required properties
PartitionKey, RowKey and Timestamp
14
Required Entity PropertiesPartitionKey & RowKey
Uniquely identifies an entityDefines the sort orderUse them to scale your application
Timestamp Read onlyOptimistic Concurrency
15
Required Properties
All entities must have the following properties:
TimestampPartitionKeyRowKey
PartitionKey + RowKey = “primary key”
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast & Furious … 2009
Action The Bourne Ultimatum
… 2007
… … … …
Animation
Open Season 2 … 2009
Animation
The Ant Bully … 2006
PartitionKey(Category)
RowKey(Title)
Timestamp
ReleaseDate
Comedy Office Space … 1999
… … … …
SciFi X-Men Origins: Wolverine
… 2009
… … … …
War Defiance … 2008
PartitionKey(Category)
RowKey(Title)
Timestamp ReleaseDate
Action Fast & Furious … 2009
Action The Bourne Ultimatum
… 2007
… … … …
Animation
Open Season 2 … 2009
Animation
The Ant Bully … 2006
… … … …
Comedy Office Space … 1999
… … … …
SciFi X-Men Origins: Wolverine
… 2009
… … … …
War Defiance … 2008
Partitions and Partition Ranges
Server BTable = Movies
[Comedy- Western)
Server ATable = Movies[Action - Comedy)
18
Server ATable = Movies
Table OperationsTable
CreateQueryDelete
EntitiesInsertUpdate
Merge – Partial UpdateReplace – Update entire entity
DeleteQueryEntity Group Transaction (new)
Creating a Table
Tables can be created and destroyed in code.Wire up your storage in the on_start method.Use a CloudTableClient to access the table.
Entity: Inherit the Required FieldsBy inheriting from TableServiceEntity we can easily store our entity in an Azure Table.
Create a Context Class
WCF Data Services needs a context class. This represents your connection to the table.
Adding an Entity
Other operations are just like normal ADO.NET Data Services,such as delete, update, etc.
Deleting an Entity
Key Selection: Things to Consider> Scalability
> Distribute load as much as possible> PartitionKey is critical for scalability> Hot partitions can be load balanced
> Query Efficiency & Speed> Avoid frequent large scans> Parallelize queries
25
Movie Listing – Solution 1> Why do I need multiple PartitionKeys?
> Account name as Partition Key > Movie title as RowKey since movie names
need to be sorted> Category as a separate property
> Does this scale?
26
PartitionKey(Account name)
RowKey(Title)
Category …
moviesonline 12 Rounds Action …
moviesonline A Bug’s Life
Animation …
100,000,000 more rows
… … …
moviesonline Office Space
Comedy …
moviesonline Platoon War …
50,000,000 more rows
… … …
moviesonline WALL-E Animation …
Server A
Movie Listing – Solution 1
> Single partition - Entire table served by one server
> All requests served by that single server> Does not scale
PartitionKey(Account name)
RowKey(Title)
Category …
moviesonline 12 Rounds Action …
moviesonline A Bug’s Life
Animation …
100,000,000 more rows
… … …
moviesonline Office Space
Comedy …
moviesonline Platoon War …
50,000,000 more rows
… … …
moviesonline WALL-E Animation …
ClientClient Request
Request
Request
Request
27
Movie Listing – Solution 2
PartitionKey(Category)
RowKey (Title)
Action Fast & Furious
… 10000 more Action movies
Action The Bourne Ultimatum
… 100000 more Action & Animation movies
Animation Open Season 2
… 100000 more Animation movies
Animation The Ant Bully
Comedy Office Space
… 1000000 more Comedy & SciFi movies
SciFi Star Trek
… 100000 more SciFi & War movies
… 100000 more War movies
War Defiance
> All movies partitioned by category> Allows system to load balance hot partitions> Load distributed> Better than single partition
ClientClient Request
Server A
Request
Request
Request
28
Server B
Request
Request
Request
Request
PartitionKey(Category)
RowKey (Title)
Action Fast & Furious
… 10000 more Action movies
Action The Bourne Ultimatum
… 100000 more Action & Animation movies
Animation Open Season 2
… 100000 more Animation movies
Animation The Ant Bully
Comedy Office Space
… 1000000 more Comedy & SciFi movies
SciFi Star Trek
… 100000 more SciFi & War movies
… 100000 more War movies
War Defiance
Key Selection: Case Study 2
> Log every transaction into a table for diagnostics> Scale Write Intensive Scenario> Logs can be retrieved for a given time
range
29
Logging - Solution 1> Timestamp as Partition Key
> Looks like an obvious choice> It is not a single partition as time moves
forward > Append only> Requests to single partition range> Load balancing does not help> Server may throttle
PartitionKey(Timestamp)
Properties
2009-11-15 02:00:01
…
2009-11-15 02:00:11
…
100000 more rows …
2009-11-17 05:40:01
…
2009-11-17 05:40:01
…
80000 more rows …
2009-11-17 12:30:00
…
2009-11-17 12:30:01
…
ApplicationsClientReques
t
Server A
Request
2009-11-17 12:30:01
…
Request
2009-11-17 12:30:02
…
Request
2009-11-17 12:30:03
…
Server B
30
Server A
Server B
PartitionKey(ID_Timestamp)
Properties
01_2009-10-12 05:10:00
…
… …
100000 more rows …
09_2009-11-15 12:31:00
…
… …
20000000 more rows …
10_2009-10-05 05:10:10
…
5000000 more rows …
… …
900000 more rows …
19_2009-11-17 12:20:02
…
ApplicationsClientReques
tReques
tReques
tReques
t
Logging Solution 2 - Distribute "Append Only”
> Prefix timestamp such that load is distributed> Id of the node logging> Hash into N buckets
> Write load is now distributed > Better throughput> To query logs in time range
> Parallelize it across prefix values
15_2009-11-17 12:30:01
…
09_2009-11-17 12:30:22
…
19_2009-11-17 12:30:10
…
01_2009-11-17 12:30:01
…
31
?Table Storage Demo
demo
?Overview Blob Storage
What is a BLOBBinary Large OBject
BLOBs in Azure & SizeEach BLOB can be up to 1TB in sizeYou can have as many containers and BLOBs as you want.Containers can be created/destroyed on the fly
Account: MovieConversion
- Job1.mpg- Job2.mpg- Header.png
Container: Originals
- Job 9.mpg- Index.docx- Job8.pdf
Container: Completed
http://<account>.blob.core.windows.net/<container>/<blobname>http://cohowinery.blob.core.windows.net/images/pic01.jpg
blobs
Blob Storage Concepts
BlobContainerAccount
sally
images
PIC01.JPG
PIC02.JPG
movies MOV1.AVI
Windows Azure Storage Services
Blobs – Provide a simple interface for storing named files along with metadata for the file
New Type of BlobPage Blob
New Blob OperationsUpdate BlobCopy BlobSnapshot BlobLease Blob
New Ways of Accessing and Serving Blob ContentCustom Domain NamesRoot Blob ContainerContent Delivery Network accessShared Access Signatures
Drives – Provides durable NTFS volumes for Windows Azure applications to use (Feb 2010)
New Features
?Overview Queue Storage
What is a queue?
Queues are used to store messagesThey are FIFO, and one wayA queue name must be lower case name, and URL friendly
MessageQueueAccount
order processing
customer ID order ID http://…
customer ID order ID http://…
cohowinery
Account, Queues and MessagesAn account can create many queues
Queue Name is scoped by the account
A Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expiration
MessagesMessage size <= 8 KBTo store larger data, store data in blob/entity storage, and the blob/entity name in the messageMessage now has dequeue count 40
Azure Boot Camp
?Q
& A
Ask your questions.
?Q
& A
Ask your questions.
What Next
Our next UG Meeting on 30th March 2011Microsoft Office Level 21, Singapore
Storage Viewer / Explorer Tools
Azure Storage Explorerhttp://azurestorageexplorer.codeplex.com/Myazurestorage.comhttps://myazurestorage.com/Cloud Xplorerhttp://clumsyleaf.com/products/cloudxplorerCerebrata Cloud Storage Studiohttp://www.cerebrata.com/products/cloudstoragestudio/
