Roman Rehak: 24/7 Database Administration + Database Mail Unleashed

Two for Tuesday1) Database Mail Unleashed

2) 24/7 Database Administration

Roman RehakPrincipal Database Architect

MyWebGrocerBurlington, VT

• Principal Database Architect/DBA Team Manager at MyWebGrocer• Independent SQL Server consultant• Specialize in

SQL Server administration, High Availability Data Access, ADO.NET, SSIS, data exchange

• Writer for Visual Studio Magazine, SQL Server Magazine, SearchSQLServer.com

• Involved in Professional Association for SQL Server (www.sqlpass.org) Speaker at 2002, 2003, 2006 PASS Community Summit Member of Program Committee the last 7 years

Track Leader for “Database and Application Development” track• DevTeach /SQLTeach Chair for SQL Server • President of Vermont SQL Server User Group• Blog: http://sqlblog.com/blogs/roman_rehak

Professional Background

http://sqlblog.com/blogs/roman_rehak

Part 1Database Mail Unleashed

SQL Mail and why it sucks

• Used in SQL Server 2000 and prior versions• Based on and relying on MAPI• Not supported on cluster instances• Many, many issues

Required Outlook installation Messages stuck in Outbox

Unless Outlook opened all the time Works with SMTP, but can hang SQL Server

If SMTP server stopped or not accessible

If you are stuck with 2000 for now

• Use xp_smtp_sendmail instead• Written by Gert Drapers• Uses SMTP only• Easy to set up• Powerful interface

HTML output Attachments

• http://www.sqldev.net/xp/xpsmtp.htm• http://mreschke.com/topic/167/SMTP+from+MSSQL+Proce

dure

http://www.sqldev.net/xp/xpsmtp.htm

http://mreschke.com/topic/167/SMTP+from+MSSQL+Procedure

http://mreschke.com/topic/167/SMTP+from+MSSQL+Procedure

Database Mail

• No More MAPI!!!• SMTP based• Asynchronous processing

Uses Service Broker

• Robust• Scalable• Fault tolerant

Database Mail architecture

• Profile This is what you specify when calling mail sp

• Accounts Contains details about SMTP servers

• System views For viewing sent emails, errors

• Security – role in msdb, public/private• SQL Agent Mail

Profile

• Holds a collection of SMTP accounts• Holds security details

Private and Public settings

• Priority for selecting order of accounts Starts with lowest, goes up Once email is sent, it keeps using the first successful account until it fails Once last successful account fails, starts at 1 again

SMTP Account• Contains settings for a single SMTP Server

Server name Port Reply-To email address

• Three authentication methods Windows Authentication Basic Authentication

Works with ISP SMTP servers Anonymous Authentication

• Can be used by multiple profiles

What happens when you send email

• Mail is queued up in mail queue in msdb• Sp_send_dbmail returns immediately

Further mail processing is asynchronous

• New row appears in sysmail_allitems Status is unsent

• DB Mail executable is notified• DB Mail attempts delivery (possibly retrying)• On success, status is updated to sent• On failure, status is updated to failed

Security

• Mail is off by default Enable with wizard or sp_configure

• Msdb has a new role – DatabaseMailUserRole Add DB users to grant send mail privileges

• Profile security Public for all users in the database Private for specific users

• Control over attachments Max size, prohibited file extensions

System views in msdb

• sysmail_allitems Shows all mail messages

• sysmail_faileditems Might be useful to monitor

• sysmail_sentitems• sysmail_unsentitems• sysmail_event_log

Shows status and error messages

• sysmail_mailattachments

Demo – Database Mail

Utilizing DB Mail in notifications

• Can be easily used with SQL Agent• Create a SQL Operator

Use valid email

• Configure SQL Agent to use DB MAIL Select Database Mail profile to be used SQL Agent NEEDS TO BE RESTARTED

• Tips for SQL Agent notifications Notify on failures ONLY Include server name in profile Set up folders and rules (labels in Gmail)

Demo – SQL Agent Mail

DB Mail Best Practices

• Configure alternate account for failover• If busy, spread over different SMTP servers

Use profiles to load balance

• Purge old emails/logs when not needed sysmail_delete_mailitems_sp (purges messages) sysmail_delete_log_sp (purges mail logs) Determine retention period Use scheduled job to purge

• Keep off if not needed


• Create a default profile• Utilize with SQL Agent• Keep security tight• Do not use multiple emails in TO, CC or BCC

Possible bug in DB Mail If one emails fails, processing stops, then retries again Successful emails up to the point of failure might get multiple copies until

processing is stopped

• Distinguish between QA/Dev and Production


• Use when it makes sense Don’t turn this your mass/spam mailer Although scalable, SQL Server has other things to do

• Good use SQL Agent alerts and notifications Emailing errors from jobs, SSIS, backend processing Data driven subscriptions Small to medium scale email in applications

Keep scalability in mind

Part 224/7 Database Administration

• High Availability Overview• SQL Server High Availability options• Case Study - HA with mirroring • Maintaining Standby Servers • Preventing downtime • Minimizing downtime• Questions

Agenda

24/7 Database Administration

• Administering databases that need to run 24/7 Extremely difficult Certain downtime required to do Windows updates, etc… Small periodic maintenance window can make a big difference

• High Availability is extremely important Different types, level of complexity, pros and cons with each Don’t believe Microsoft marketing

• Requires planning - drills are crucial “Si vis pacem, para bellum” Things unexpected do happen, but we can learn from them Need to have monitoring and alerts in place

• Having a talented and dedicated team is crucial

Successful High Availability components

• High availability options at the SQL Server level• High availability at the hardware level• Monitoring• Alerts and notifications• Trained and responsive team• Emergency response procedures• Emergency drills• Sufficient documentation


Agenda

SQL Server High Availability

• The true HA does not exist yet• SQL Server fundamentally remains a single point of failure• Different options exist

Each has prons and cons Failing database(s) over is easy, but SQL Agent jobs need attention too SQL Server Clustering got it close, but there were too many issues to make

it hard to run reliably

• What we need is a true, failure transparent cluster• In the meantime

Learn what’s available Pick the right technology/tools

Native Replication

• When to use Expertise in the house Publishing will not affect data availability

Deadlocks while publishing Published tables get locked up

• Suitable Data redundancy Less suitable as an application backend

• If you use it Implement alerts when failures occur Respond quickly Protect subscribers with triggers, read-only privileges

Restored Database Copies

• Simple and effective way of spreading data load• Suitable when data latency is not super high• Added benefit – testing your backups• Good uses

Reporting server Source for Data Warehouse extracts Application – use separate connection string for data older that X number of

hours

• You may need to kick off custom SQL after a restore Logins, permissions, additional objects, etc…

• It can provide partial High Availability (order history, …)

SQL Server Clustering

• Multiple physical nodes create a virtual node• Some learning curve but it has become easier• One huge advantage – exposed by a virtual name,

regardless of what node it’s running on• Failover is transparent• You lose existing connections• No changes in connection strings needed• Scheduled jobs continue to work• Good hardware recommended

AlwaysOn

• New in SQL Server 2012• Powerful but complex• Requires Enterprise license for both SQL Server and

Windows to get all features• Improvements

Database availability groups Exposed as a single endpoint for connection strings More than one replica Data on replicas can be used for read-only access

• This may not work for small companies

Database Mirroring

• Database is mirrored to another server• In case of an issue, you can failover to mirror server

Manual failover Automatic failover

• Transactions applied to mirror High Performance – asynchronous High Safety - synchronous

• Not server level Need to maintain logins, jobs, etc

• Database not available on mirror But, you can create a snapshot Possibly HA/load balancing option


Agenda

Database Mirroring Demo

Database Mirroring Tips• Script it out

The wizard is OK, but tedious when doing 10+ databases

• Increase the default timeout 10 seconds not enough ALTER DATABASE Foo SET PARTNER TIMEOUT 120

• Do not mirror to an SSD drive if possible Replaying of transaction log different than on the primary 6 times more I/O

• Set up alerts Several events, you probably want to know if a failover occurred

• Set up backups on both sides Use “Ignore databases not online” in the backup task (Maintenance Plans)

Intermission


Agenda

Dual-server deployment

• Needed anywhere you use a standby machine (AlwaysOn, Database Mirroring, SQL Server Clustering, Log Shipping)

• Deploy to each SQL Server instance SQL Logins Scheduled jobs Linked Servers Note – N/A for clustering

• Deploy to each physical computer Powershell scripts SQL scripts Executables/DLL components SSIS packages

Dual-server deployment tools• Scripting logins

Script out logins on primary, apply to standby servers Use my script to preserve SIDs for SQL Server logins Make sure you have fast access to passwords if needed

• Synchronizing scheduled jobs Many different options out there You can use my SSIS package Generates scripts multiple servers, saves to a folder Typically you can ignore replication and maintenance plans jobs

Utilize categories to manage/script only what you need

• Use automation Sqlcmd Powershell

Scripting SQL Agent Jobs Demo


Agenda

Preventing downtime• Extensive monitoring

Performance (3rd party tools) Low disk space Transaction logs growing too large

Failed replication Paused/Suspended mirroring Not setting up log backups Disabling log backups

• Set up alerts Failed jobs, mirroring disconnects, replication hiccups Page only when needed

• Distribute email notification handling within the team • Do not notify of success unless you really need it

Preventing downtime

• Utilize Policy Management Many possible types of alerts can be easily created Examples

Email us when no full backup was done within last 25 hours Email us when no log backup was done within last 2 hours Many other possibilities

• Invest in good hardware EBay is not your friend

• Run DBCC periodically• Use tools for database access, not direct SQL

If you need direct updates, train people to COMMIT

Preventing downtime

• DBA instinct• Learn from your incidents• Build checklists for common procedures

Example – Enable what you disabled, etc..

• Build a KB/Wiki/Whatever for your company• Cross-train the team members

DBA Mirroring At least two people to be SME, ideally more than that

Warning - SSD/RAMSAN drives

• Great for speed increase• Mainly for READS, much less for WRITES• Not suitable for transaction logs• Potential issues

Single point of failure The number of good sectors does decrease Database mirroring causes fast wear&tear

• If you use them Consider implementing an array of SSDs


Agenda

Minimizing downtime

• Pager is very important for quick response Extra compensation makes a difference Per week or per incident

• Sufficient documentation Who uses what, when, how, emergency contacts, downtime implications,

etc… Development team input required

• Demand regular maintenance nights Upgrades, re-indexing, table locking ops, etc

• Have “Maintenance pages” ready to go Takes pressure off somewhat

• Tell execs and managers to back off

Minimizing downtime• Utilize automation to rebuild things

Wizards are great but slow Scripting is your friend here Examples – rebuild database mirroring, replication

• Data Center tips Make sure multiple employees are familiar with the data center Utilize “Remote Hands” service Line up “Remote Hands” for SQL Server restarts and reboots If you lose power – BRING DOMAIN CONTROLLERS ONLINE FIRST!

• Develop procedures for possible SQL Server issue SUSPECT database Losing TEMPDB data or log drive Losing MASTER database

Questions?

• Email: [email protected]

Technology

Roman Rehak: 24/7 Database Administration + Database Mail Unleashed