Development Best Practices and Patterns for Using Microsoft SQL Azure DatabasesTony PetrossianPrincipal Program Manager

SVC03

Agenda> Service Recap> Server Management> Security and Access Control> Connectivity and Tracing> Latency and Connectivity Patterns> Dealing with large amounts of data> Getting Data In and Out

SQL Azure Recap

> SQL Azure provides capacity on demand> Create a “server”> Create database within your server

> Server lifecycle controlled via Portal> Portal login using LiveID

> Database lifecycle via normal TSQL> Authenticate using ‘SQL login’ over

TDS+SSL

SQL Azure Network TopologyApplicatio

nInternetAzure Cloud

LB

TDS (tcp)

TDS (tcp)

TDS (tcp)

Applications use standard SQL client libraries: ODBC, ADO.Net, PHP, …

Load balancer forwards ‘sticky’ sessions to TDS protocol tier

Security Boundary

SQL SQL SQL SQL SQLSQL

Gateway

Gateway

Gateway

Gateway

Gateway

Gateway

Scalability and Availability: Fabric, Failover, Replication, and Load balancing

Gateway: TDS protocol gateway, enforces AUTHN/AUTHZ policy; proxy to backend SQL

Servers> Each SQL Azure server provides

> Geo-location (has a unique DNS name)> A zone for administration policy> A point of billing and reporting aggregation

> Where should I create my server?> Best practice: co-locate server with Windows

Azure app role (if using) to reduce latency

> When should I create a new server?> Trade off between geo/admin/billing

Server Management> Through the Portal

> Add/Drop server> Establish admin credentials> View usage reports> Network access configuration

> Through the Master Database> Fine-tune firewall settings through code> User logins> Usage and metrics reporting (billing)> Create/Drop databases

Portal

Master DB

User DB

User DB

User DB

HTTP

TDS

SQL Azure Server

Server: Network Access Control

> Each server defines a set of firewall rules> Determines access policy based on client IP> By default, there is NO ACCESS to server

> Controlled using > TSQL API against Master DB:

sys.firewall_rules, sys.sp_set_firewall_rule, sys.sp_delete_firewall_rule

> Portal UXID Nam

eStart IP

End IP Create Modify

1 Office 12.1.2.0

12.1.2.255

2009-09-18 …

2009-09-18 …

2 Home 12.2.2.5

12.2.2.5 2009-09-20 …

2009-09-21 …

Security: AUTHN and AUTHZ> SQL Azure uses SQL authentication

(UID/PWD)

> Authorization model fully compatible with SQL

> Admin roles has permission for> CREATE/DROP database > CREATE/DROP/ALTER login> GRANT/REVOKE rights> Modifying server firewall settings

Server: Billing and Reportingsys.bandwidth_usage: usage in KB

sys.database_usage: instance count by SKU

Time Database

Direction

Class Period Quantity

2009-09-17 19:00

TPCH Egress Internal

Peak 55598

2009-09-17 19:00

TPCH Ingress Internal

Peak 76026

… … … … … …

Time SKU Quantity

2009-09-17 19:00

Web 1

2009-09-17 19:00

Business 10

… … …

Connecting to SQL Azure> SQL Azure connection strings follow normal SQL syntax

> Except for an unusual username format

> Format of username for authentication:> ADO.Net:

Data Source=server.database.windows.net;User ID=user@server;Password=password;...

> ODBC:Driver={SQL Server Native Client 10.0}; Server=server.database.windows.net; Uid=user@server;Pwd=password;...

> Applications connect directly to a database> “Initial Catalog = <db>” in connection string> No support for context switching (no USE <db>)

Built-in Connection Management> Connections may drop due to:

> Network connectivity blips> Idle or long running transactions

> Idle > 5 minutes> Long running transactions > 5 minutes

> Throttling (taking too many resources)> Measured by IO load and CPU utilization

> Database failover activity> Load balancing used to ensure ‘resource fairness’

> DOS protection may deny connectivity:> If too many failed connection attempts are made> Your server is not impacted by such attacks

Recommendations

Application Design Topics> Most-applicable SQL Best Practices

> Connection Pooling> Query Parameterization> Batching

> Scaling with data and load> Sharding> Building copies

> Deploying and uploading data> Bulk copy> SSIS> Sync

Use Pooled Connections

// When pooling, use connection and return immediately// Do not hold for a long time – pool ensure fast turnaround// one second useusing (SqlConnection conn = new SqlConnection(…)){ conn.Open(); using (SqlCommand cmd = conn.CreateCommand()) { cmd.CommandText = …; … }}using (SqlConnection conn = new SqlConnection(…)){ conn.Open(); …

Increases efficiency by removing re-login

Connections: Retry on failure> Connections can drop for variety of reasons

> Idleness> Transient (network) errors> Intentional throttling

> First step: reconnect immediately> Handles idleness- and transient-disconnects

> Gateway handles connection retry for app> Connections attempted for ~30s before failure

> What to do on connection failure?> Wait (10 seconds), then retry > Change your workload if throttled

> Server health can be checked via Portal> TSQL APIs will come in later releases

Connection Pattern

while (true){ using (SqlConnection connection = new SqlConnection(connStr)) { try { connection.Open(); using (SqlCommand cmd = connection.CreateCommand()) { cmd.CommandText = @"SetBCPJobStartTime"; cmd.CommandType = CommandType.StoredProcedure; cmd.Parameters.Add(new SqlParameter(@"@BCPJobId", BCPJobId)); cmd.ExecuteNonQuery(); } } catch (Exception exc) { // deal with error } } // more stuff // …..}

Tracing Connectivity Problems> Each session assigned a unique

‘sessionId’> Tracks session state and service errors

> Retrievable from CONTEXT_INFO()> Save this with each new connection

> If you need support, support personnel will greatly appreciate that value

Tracing Helper Pattern

Guid? sessionId = null;using (SqlConnection conn = new SqlConnection(…)){ // Grab sessionId from new connection using (SqlCommand cmd = conn.CreateCommand()) { conn.Open(); cmd.CommandText = "SELECT CONVERT(NVARCHAR(36), CONTEXT_INFO())"; sessionId = new Guid(cmd.ExecuteScalar().ToString()); }

// continue processing ...}

Maximize your performance> Batching: push logic to Server

> Use stored procedures and batching> Limit number of round trips to server

> Example: batch 10 statements vs. 10 round-trips

> Parameterized queries> Parameterize queries (limits compiles)> Declare all parameters, type and length> Lack of parameter size leads to cache

bloat

Declare Parameter Lengths!

// Length inferred: leads to cache bloat cmd.CommandText = "SELECT c1 FROM dbo.tbl WHERE c2 = @1";cmd.Parameters.Add("@1", SqlDbType.NVarChar).Value = "1";…cmd.Parameters.Add("@1", SqlDbType.NVarChar).Value = "22";

// Length supplied: no cache bloat cmd.CommandText = "SELECT c1 FROM dbo.tbl WHERE c2 = @1";cmd.Parameters.Add("@1", SqlDbType.NVarChar, 128).Value = "1";

(@1 nvarchar(1)) SELECT c1 FROM dbo.tbl WHERE c2 = @1(@1 nvarchar(2)) SELECT c1 FROM dbo.tbl WHERE c2 = @1

(@1 nvarchar(128)) SELECT c1 FROM dbo.tbl WHERE c2 = @1

Maximize elasticity benefits> SQL Azure balances databases across machines

> Divide your data into smaller chunks> Makes for better load-balancing> Ensures we can place your data on the most-

appropriate servers

> Using shorter transactions> Ensures we can respond to issues faster> Avoid impacting others sharing the same box

> Thus avoiding being throttled by SQL Azure

SQL Azure ClusterNode

14Node

19Node

21Node

99 Node 2Node

76

Shards: Managing Lots of Data

App

Server

Master DB

DB1

DB2

Node 33

DB3

DB4

DB5

DB6

SQL Azure ClusterNode

14Node

19Node

21Node

99 Node 2Node

76

Copies: Managing Lots of Read Access

App

Server

Master DB

DBC1

DBC2

Node 33

DBC3

DBC4

DBC5

DBC6

LB

Migrating Applications to SQL Azure

Getting Data In and Out> SQL Azure supports standard SQL data

import and export patterns

> Use bulk loading patterns where possible> BCP – console .EXE bulk load/export tool> SSIS – SQL integration server> Bulk APIs in ODBC and ADO.Net

> SQL Azure supports data synchronization> With on-premises DBs and client stores

Getting Large Data into SQL Azure> Always good advice:

> Break batches up into smaller, consumable chunks

> Add retry and tracing logic to ensure robust resume in face of failures

Data Import: ADO.Net Bulk Copy API

// Bulk data importusing (SqlBulkCopy bulk = new SqlBulkCopy(new SqlConnection(conn)) { DestinationTableName = "dbo.data", BatchSize = 2000, // Transaction size (length) BulkCopyTimeout = 10000, // Transaction timeout NotifyAfter = 1000, // Progress callback }){ bulk.SqlRowsCopied += new SqlRowsCopiedEventHandler( myProgressCallback); bulk.WriteToServer(sourceDataReader);}

SSIS

demo

Data Export/Import: BCP.EXE

// BCP exampleSET SRV=somesrv.database.windows.netSET LOGIN=mylohin@somesrvSET PW=somethingSET S_DR=C:\flats

bcp TPCH2.dbo.supplier in %S_DR%\supplier.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp TPCH2.dbo.nation in %S_DR%\nation.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp TPCH2.dbo.region in %S_DR%\region.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp TPCH2.dbo.customer in %S_DR%\customer.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp TPCH2.dbo.part in %S_DR%\part.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|“

bcp TPCH2.dbo.supplier out %S_DR%\supplier.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp TPCH2.dbo.nation out %S_DR%\nation.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp TPCH2.dbo.region out %S_DR%\region.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp TPCH2.dbo.customer out %S_DR%\customer.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"bcp TPCH2.dbo.part out %S_DR%\part.tbl -c -U %LOGIN% -P %PW% -S %SRV% -t "|"

End to End Demo: Azure BCP Loader

> Load blobs to Azure> BCP from Azure worker> Reduced latency

improved throughput

Browser

WorkerRole

Blobs

Target DB

Jobs

BCP

PutBlob()NewJob()

GetJob()GetBlob()

WebRole

Summary> Many SQL Server patterns apply to SQL

Azure> Use SQL best practices wherever possible

> Patterns discussed:> Connectivity (to database, not server)> Tracing and support> Batching, Pooling and Parameterization> Getting data in and out

YOUR FEEDBACK IS IMPORTANT TO US! Please fill out session evaluation

forms online atMicrosoftPDC.com

Learn More On Channel 9> Expand your PDC experience through

Channel 9

> Explore videos, hands-on labs, sample code and demos through the new Channel 9 training courses

channel9.msdn.com/learnBuilt by Developers for Developers….

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.