71
Tivoli Storage, IBM Software Group Potpourri du Backup Andy Raibeck Oxford University TSM Symposium 2007 St. Catherine’s College, 25 – 27 September © 2007 IBM Corporation

Tivoli Storage, IBM Software Group

  • Upload
    others

  • View
    8

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Potpourri du Backup

Andy RaibeckOxford University TSM Symposium 2007St. Catherine’s College, 25 – 27 September

© 2007 IBM Corporation

Page 2: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

TrademarksTrademarks

The following terms are trademarks of International Business gMachines Corporation in the United States, other countries, or both:

• AIX• IBM• Tivoli

Other company, product, and service names may be trademarks or service marks of othersservice marks of others.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United St t d th t iStates and other countries.

© 2007 IBM Corporation2 Potpourri du Backup

Page 3: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

IntroductionIntroduction

There is a potpourri of backup techniques available in e e s a potpou o bac up tec ques a a ab eTSM

Probably more to come in the futureProbably more to come in the future

One size does not fit all

Why can’t TSM decide for me?

© 2007 IBM Corporation3 Potpourri du Backup

Page 4: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

What is covered in this presentationWhat is covered in this presentation

The various backup techniques offered by TSM Backup-Archive p q y pClient:– What you get vs. what you give up with different technologies and

variationsvariations

– Decision points

Provide qualitative advice on evaluating different technologies andProvide qualitative advice on evaluating different technologies and variations– What tools are available today to help with these decisions

© 2007 IBM Corporation4 Potpourri du Backup

Page 5: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

What is NOT covered in this presentationWhat is NOT covered in this presentation

How to implement:o to p e e t– Not a tutorial in option syntax

– No dsm.opt or dsm.sys examples!No dsm.opt or dsm.sys examples!

Other backup or archive considerations

Recovery or retrieve considerations

© 2007 IBM Corporation5 Potpourri du Backup

Page 6: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

AgendaAgenda

Backup techniques– What was available before ADSM/TSM came on the scene– TSM Progressive incremental backup model and its derivatives– Image backupImage backup

How to determine which backup technique best fits– What problem are you trying to solve– Windows– Other operating systems

© 2007 IBM Corporation6 Potpourri du Backup

Page 7: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

AgendaAgenda

Backup techniques– What was available before ADSM/TSM came on the scene– TSM Progressive incremental backup model and its derivatives– Image backupImage backup

How to determine which backup technique best fits– What problem are you trying to solve– Windows– Other operating systems

© 2007 IBM Corporation7 Potpourri du Backup

Page 8: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Other Backup Products TSM

Long restore timeP di tili ti

No unnecessary backups Long restore timeP di tili ti

Other Backup Products TSM

Poor media utilizationLots of copies of unchanged dataHigh network utilizationHigher impact to application server

Client data consolidated on a few tapesLower network utilizationLow impact to application

Poor media utilizationLots of copies of unchanged dataHigh network utilizationHigher impact to application server

Full + Incremental

Full Backup

Progressive incrementalBase BackupFull Backup

Full + Incremental Full + Differential

Incremental

Differential

Incremental

Incremental

BAIncremental

Incremental

Differential AutomatedTapereclamation

Full Backup

Incremental

Full Backupreclamation

IncrementalIncremental

Incremental Tape reclamationCollocation

© 2007 IBM Corporation8 Potpourri du Backup

Collocation

Page 9: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

AgendaAgenda

Backup techniques– What was available before ADSM/TSM came on the scene– TSM Progressive incremental backup model and its derivatives– Image backupImage backup

How to determine which backup technique best fits– What problem are you trying to solve– Windows– Other operating systems

© 2007 IBM Corporation9 Potpourri du Backup

Page 10: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Progressive Incremental BackupProgressive Incremental Backup

1 23

DB

1

2

2

1 Cli t i f t i f fil t

4

1. Client queries server for current view of file system

2. Server returns list of active versions for entire file system

3 Client scans and compares with local file system3. Client scans and compares with local file system

4. Client backs up new and changed files

© 2007 IBM Corporation10 Potpourri du Backup

Page 11: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Progressive IncrementalProgressive Incremental

a.k.a Full Progressive Incremental or Full Incremental or g“Incremental Forever”

What you gety g– Single-instance store

• Technically progressive incremental backup is a form of SIS (not a de-dup though)dup though)

– Easier restore• No concept of applying incremental or differential changes to full

© 2007 IBM Corporation11 Potpourri du Backup

Page 12: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Progressive IncrementalProgressive Incremental

When to use thise to use t s– Default technology in TSM

– Hot spots:Hot spots:• Memory

– Number of active backup versions– Length of file nameg

• Run time– Biggest factor is file system scan time

© 2007 IBM Corporation12 Potpourri du Backup

Page 13: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Progressive Incremental DerivativesProgressive Incremental Derivatives

Memory efficient backupe o y e c e t bac up

Memory efficient disk caching

Virtual mount points

Incremental by date

File lists

J l b d b kJournal-based backups

© 2007 IBM Corporation13 Potpourri du Backup

Page 14: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Memory EfficientMemory Efficient

1 23 5

DB

1

2

25

1 Cli t i f t i f fi t di t

41. Client queries server for current view of first directory

2. Server returns list of files in directory (db query)

3. Client scans and compares with local file system

4. Client backs up new and changed files

5. Client queries server for current view of next directory, etc.

© 2007 IBM Corporation14 Potpourri du Backup

Page 15: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Memory Efficient BackupMemory Efficient Backup

What you get– Smaller backup memory footprint

What you don’t getMemory savings in directories with large number of objects– Memory savings in directories with large number of objects

What you give up– More network chit-chat– Increased backup run time

When to use thisIncremental backups run out of memory– Incremental backups run out of memory

– Can get rough estimate on memory usage

© 2007 IBM Corporation15 Potpourri du Backup

Page 16: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Memory Efficient Disk CachingMemory Efficient Disk Caching

1 23

DB

1

2

2

4

1. Client queries server for current view of file system

2. Server returns list of files for entire file system; client stores inventory on disk rather than in memoryy

3. Client scans and compares with local file system

4. Client backs up files

© 2007 IBM Corporation16 Potpourri du Backup

Page 17: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Memory Efficient Disk CachingMemory Efficient Disk Caching

How it works– Same as progressive incremental except …– B-A client can store TSM Server db queries on disk instead of in memory

What you getWhat you get– Even smaller memory footprint if regular memoryefficient backup doesn’t

reduce memory

Wh t iWhat you give up– Processing time (takes a bit longer to store and retrieve information from

disk)

When to use this– Incremental backup running out of memory and memoryefficient backup

not sufficient Need individual file restore– Journal based backup technology is not available on your platform (note

you might need this just to get the first backup done for journal based backups).

© 2007 IBM Corporation17 Potpourri du Backup

Page 18: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Virtual Mount PointVirtual Mount Point

/home

/home/1 /home/2

/home/2

Large file system logically partitioned

Managed as separate file spaces on TSM Server

© 2007 IBM Corporation18 Potpourri du Backup

Page 19: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Virtual Mount PointVirtual Mount Point

How this works– Instead of mapping entire file system to single namespace on TSM

Server (file space), user can define– What you get– Progressive incremental only has to work against subset of file system

thus reducing memory requirements

What you don’t gety g– Relief from directories with large number of objects

What you give up– Centralized management of file system on TSM Server– Virtual mount points are static and can’t be migrated

When to use thisWhen to use this– Large, balanced (number of directories and objects) UNIX and Linux file

systems can be efficiently divided

© 2007 IBM Corporation19 Potpourri du Backup

Page 20: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Incremental by DateIncremental by Date

13

DB

1

2

4

1. Client queries server last backup of entire file system

2. Server returns timestamp of last backup (end) of entire file system

3 Client scans and compares with local file system3. Client scans and compares with local file system

4. Client backs up files

© 2007 IBM Corporation20 Potpourri du Backup

Page 21: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Incremental By DateIncremental By DateWhat you get– Reduced time determining which files have changed– Removing processing time on server to query database – Removing network traffic to communicate results

What you give upFl ibilit f b k ti t b d i b k f th ti fil t– Flexibility on scope of backup operation; you must be doing backups of the entire file system

– File backups where changes do not affect date (attribute, mode, ACL)• Rename, copy, move, security change

– Expiration of deleted files– Policy rebinding– Entire file system still needs to be scanned– No free pass for having client and server clocks out of synch

May not be able to use when client and server are in different time zones– May not be able to use when client and server are in different time zones.

When to use this– Not meeting backup windows and …

File system changes purely additive or changes e g code repository or data that is basically– File system changes purely additive or changes, e.g., code repository or data that is basically “additive” (think of data like your family photos)

Can be effective when doing weekly (or periodic) full incremental backupsRead Chapter 9 in the User’s Guide for more critical information on this option

© 2007 IBM Corporation21 Potpourri du Backup

p p

Page 22: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

File List BackupFile List Backup

2application

1DB3

1

1 A li ti t li t f fil f b k1. Application creates list of files for backup

2. Application passes list of files to client

3. Client backs up files (SELECTIVE backup)

© 2007 IBM Corporation22 Potpourri du Backup

Page 23: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

File listsFile lists

What you gety g– Selective backup eliminates the query of the TSM Server db and

scan of local file system

What you give up– Time – you have to find some mechanism to feed the file list– Implicit file specifications – file list will not accept wild cards or

directory recursion

When to use thisWhen to use this– Not meeting backup windows and …

File system changes are predictable e g an application is– File system changes are predictable, e.g., an application is creating the changes and it would be easy to interface into the application to get that list of changes

© 2007 IBM Corporation23 Potpourri du Backup

Page 24: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Journal Based BackupJournal Based Backup

2journal

1DB3

1

1 Journal monitors file system for changes1. Journal monitors file system for changes

2. Client queries journal for file system changes

3 Cli t b k fil3. Client backs up files

© 2007 IBM Corporation24 Potpourri du Backup

Page 25: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Journal Based BackupJournal Based Backup

What you get– Time to determine which files have changed is greatly reduced

What you don’t get– Relief from doing a full progressive (full) incremental backup against a file system on someRelief from doing a full progressive (full) incremental backup against a file system on some

periodic basis:• Initial backup to enable the journal• If a discrepancy is found between journal and TSM Server database• If the velocity of changing files is high enough to cause a notification buffer overrun• If the journal service is stopped and restarted• In general, it is a good idea to do periodic progressive incremental backups

When to use thisN t ti b k i d d– Not meeting backup window and …

– Small number of files (< 1,000,000) and small number of changes between backups (<1,000,000)

– Large number of objects (<10 000 000) with 10-15% change rate try JBB (should emphasize a– Large number of objects (<10,000,000) with 10-15% change rate try JBB (should emphasize a low change velocity, too… large numbers of changes over a short timeframe can cause notification buffer overflows)

– Large number of objects + large number of changes will stress JBB … not a good fit

© 2007 IBM Corporation25 Potpourri du Backup

Page 26: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Adaptive Sub-file BackupAdaptive Sub file Backup

1cache

3

DB1

1cache

2 3

1. Client determines the file is a candidate for backup

2. Client backs up files

3 During data movement client determines if there is enough information stored in3. During data movement, client determines if there is enough information stored in the local cache to determine which sub-file parts have changed

© 2007 IBM Corporation26 Potpourri du Backup

Page 27: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Adaptive Sub-file BackupAdaptive Sub file Backup

What you get– Faster throughput– Reduced storage pool consumption

What you give upWhat you give up– Local cache space (in the tens of megabytes)– A bit of CPU– Restore time (due to restore of base + delta)– Possible out-of-space conditions, if disk space is tight during restore, due to how

file is reconstructed from base + delta

When to use this– Network constrained– Small file profiles (limited to file size < 2 GB)Small file profiles (limited to file size 2 GB)

© 2007 IBM Corporation27 Potpourri du Backup

Page 28: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

AgendaAgenda

Backup techniques– What was available before ADSM/TSM came on the scene– TSM Progressive incremental backup model and its derivatives– Image backupImage backup

How to determine which backup technique best fits– What problem are you trying to solve– Windows– Other operating systems

© 2007 IBM Corporation28 Potpourri du Backup

Page 29: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Image BackupImage Backup

DB1

1. Client sends logical block image of file system to server

© 2007 IBM Corporation29 Potpourri du Backup

Page 30: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Image BackupImage Backup

No mapping between file and block

Off-line and static backups vs. on-line backups

What you getF t b k– Faster backups

– No scan time to determine what has changed– Faster overall data movement

What you give up– Ability to restore individual files directly from TSM Server (we know, we know…)

• You would need to restore image to alternate location and pull data directly using Explorer g p y g por equivalent.

© 2007 IBM Corporation30 Potpourri du Backup

Page 31: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Image BackupImage Backup

Image backup– Backup at logical volume level– Ability to take image + incremental backups to provide single file restoreOff-line image backupg p– Volume mounted read-only– Available for AIX, Windows, Linux x86, Solaris and HP– Exploited best by flashcopy operationsp y py pOn-line image (dynamic) backup– Volume left on-line

Fuzzy backup of changing data– Fuzzy backup of changing dataOn-line image backup using snapshot– Volume left on-line– Image backup taken at single point-in-time (“crash consistent”)– Available for Windows and Linux x86

© 2007 IBM Corporation31 Potpourri du Backup

Page 32: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Image + Incremental by DateImage Incremental by Date

421

DB3

2

f f

51. dsmc backup image sends logical block image of file system to server2. Upon subsequent dsmc backup image –mode=incremental, client queries server

last backup of entire file system3 S t ti t f l t b k f ti fil t3. Server returns timestamp of last backup of entire file system4. Client scans and compares with local file system5. Client creates transactions of files for backup

© 2007 IBM Corporation32 Potpourri du Backup

Page 33: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Image + Incremental by Date Restore ScenarioImage Incremental by Date Restore Scenario

1DB

2

1

31. Client request incremental image restore

2. Server returns the base image

3 Server returns additional files which need to be applied to the base image to3. Server returns additional files which need to be applied to the base image to satisfy the recovery point

© 2007 IBM Corporation33 Potpourri du Backup

Page 34: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Image + Incremental By DateImage Incremental By Date

What you getat you get– Benefit of image backup plus some individual file protection of

files that are changing

What you give up– See Incremental by DateSee Incremental by Date

– No reconciliation of deleted files

© 2007 IBM Corporation34 Potpourri du Backup

Page 35: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Image + Progressive Incremental BackupImage Progressive Incremental Backup

2 341

DB

2

3

3

51. dsmc backup image sends logical block image of file system to server2. Upon subsequent dsmc incremental, client queries server for current view of file system3. Server returns list of files for entire file systemy4. Client scans and compares with local file system5. Client backs up changed files

© 2007 IBM Corporation35 Potpourri du Backup

Page 36: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Image + Progressive Incremental Restore ScenarioImage Progressive Incremental Restore Scenario

1

DB2

33

4x

1. Client requests incremental image restore

2. Server returns the base image

3 Server returns additional files which need to be applied to the base image to3. Server returns additional files which need to be applied to the base image to satisfy the recovery point

4. Server optionally returns the list of files which need to be deleted from the base image to satisfy the recovery point

© 2007 IBM Corporation36 Potpourri du Backup

g y y p

Page 37: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Image + Progressive IncrementalImage Progressive Incremental

What you gety g– Benefit of image backup restore for improving RTO

– Benefit of progressive incremental backups for better RPO

What you give up– Additional time to take periodic image backups

– Storage space at TSM Server

© 2007 IBM Corporation37 Potpourri du Backup

Page 38: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

DisclaimerDisclaimer

The performance data contained in the following slides e pe o a ce data co ta ed t e o o g s deswas measured in a controlled environment. Results obtained in other operating environments can vary significantly, depending on factors such as system workload and configuration. Accordingly, this data does not constitute a performance guarantee or warrantynot constitute a performance guarantee or warranty.

© 2007 IBM Corporation38 Potpourri du Backup

Page 39: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Improved incremental backup memory usageImproved incremental backup memory usage

Incremental backup, 1 million files workload, Win2003 server, 1 WinXP client, LAN, TCP/IP, Disk storage poolIBM x360, 4-way 1.6GHz Xeon MP server, 6 GB, IBM PC 1-way 3GHz Pentium 4 client, 512 MB, Compression No,

Encryption No Storage pool CRC No Bufpoolsize 131072 Txngroupmax 256 TXNBytelimit 2097152 NTFS filespace

DescriptionElapsed time (seconds)

0% changed data 5% changed data 20% changed data

Full incremental backup, memoryeff diskcachem, default db key size 3138 4102 5742

Encryption No, Storage pool CRC No, Bufpoolsize 131072, Txngroupmax 256, TXNBytelimit 2097152, NTFS filespace

Full incremental backup, memoryeff diskcachem, optimal db key size 2002 2850 4631

Full incremental backup, memoryefficient no 939 1620 4320

Full incremental backup, memoryefficient yes 813 1329 2479

Incremental-by-date incremental backup 707 1384 4699

Journal-based incremental backup 2 1995 6133

The disk cache method writes server inventory data to a client disk file

Tivoli Storage Manager Version 5.4.0 Incremental Backup PerformanceWindows XP Client (3GHz, 512 MB RAM)

1 million 1KB files workload

The disk cache method uses less than 20 MB of memoryThe disk cache method is generally slower than other methods

3000

4000

5000

6000

7000

ed ti

me

(sec

)

Full incremental backup,memoryefficientbackup diskcachemethod,default db key size

Full incremental backup,memoryefficientbackup diskcachemethod,optimal db key size

Full incremental backup,memoryefficientbackup no

0

1000

2000

3000

0% 5% 20%

Percent changed dataEl

apse Full incremental backup,

memoryefficientbackup yes

Incremental-by-date incremental backup

Journal-based incremental backup

© 2007 IBM Corporation39 Potpourri du Backup

Page 40: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Improved incremental backup memory usageImproved incremental backup memory usage

Incremental backup, 1 million files workload, Win2003 server, 8 WinXP clients, LAN, TCP/IP, Disk storage poolIBM x360, 4-way 1.6GHz Xeon MP server, 6 GB, IBM PC 1-way 3GHz Pentium 4 clients, 512 MB, Compression No,

Encryption No Storage pool CRC No Bufpoolsize 131072 Txngroupmax 256 TXNBytelimit 2097152 NTFS filespace

DescriptionElapsed time (seconds)

0% changed data 5% changed data 20% changed data

Full incremental backup, memoryeff diskcachem, default db key size 4142 5520 8551

Encryption No, Storage pool CRC No, Bufpoolsize 131072, Txngroupmax 256, TXNBytelimit 2097152, NTFS filespace

Full incremental backup, memoryeff diskcachem, optimal db key size 2969 4168 8120

Full incremental backup, memoryefficient no 1975 3436 7241

Full incremental backup, memoryefficient yes 1086 2177 4763

Incremental-by-date incremental backup 812 2189 5420

Journal-based incremental backup 4 2074 5746

For all tests measured with 8 WinXP clients, the disk cache method was slower than the ' ffi i tb k ' th d

Tivoli Storage Manager Version 5.4.0 Incremental Backup Performance

8 Windows XP Clients (3GHz, 512 MB RAM)1 million 1KB files workload'memoryefficientbackup yes' method

For 5% changed data, this elapsed time difference was measured at +1991 seconds, or 91% longer

4000

5000

6000

7000

8000

9000

ed ti

me

(sec

)

Full incremental backup,memoryefficientbackup diskcachemethod,default db key size

Full incremental backup,memoryefficientbackup diskcachemethod,optimal db key size

Full incremental backup,memoryefficientbackup no

0

1000

2000

3000

4000

0% 5% 20%

Percent changed dataEl

apse Full incremental backup,

memoryefficientbackup yes

Incremental-by-date incremental backup

Journal-based incremental backup

© 2007 IBM Corporation40 Potpourri du Backup

Page 41: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Improved incremental backup memory usageImproved incremental backup memory usage

Incremental backup, 21 million files workload, Win2003 server, Win2003 client, LAN, TCP/IP, Disk stgpoolIBM x360, 4-way 1.6GHz Xeon MP server, 6 GB, IBM x346, 2-way 3.4GHz client, 4GB, Compression No, Encryption

No Storage pool CRC No Bufpoolsize 131072 Txngroupmax 256 TXNBytelimit 2097152 NTFS filespace

DescriptionElapsed time (seconds)

0% changed data 5% changed data 20% changed data

Full incremental backup, memoryeff diskcachem, default db key size n/a 33699 35536

No, Storage pool CRC No, Bufpoolsize 131072, Txngroupmax 256, TXNBytelimit 2097152, NTFS filespace

Full incremental backup, memoryeff diskcachem, optimal db key size 18654 21678 24680

Full incremental backup, memoryefficient no Failed Failed Failed

Full incremental backup, memoryefficient yes 13760 18394 23310

Incremental-by-date incremental backup 11262 14679 20902

Journal-based incremental backup 2 10916 Failed

The 'memoryefficient no' method fails for this workload because it needs to use more virtual memory than is

il bl t 32 bit li t

Tivoli Storage Manager Version 5.4.0 Incremental Backup Performance

Windows 2003 (32bit) Client (3.4GHz, 4GB RAM)21 million 1KB files workloadavailable to a 32 bit client

The journal-based backup method fails when there are lots of changed filesThe disk cache method with optimal key size is only 18% slower than the 'memoryefficient yes' method at

20000

25000

30000

35000

40000

ed ti

me

(sec

)

Full incremental backup,memoryefficientbackup diskcachemethod,default db key size

Full incremental backup,memoryefficientbackup diskcachemethod,optimal db key size

Full incremental backup,memoryefficientbackup no

18% slower than the memoryefficient yes method at 5%

0

5000

10000

15000

0% 5% 20%

Percent changed dataEl

apse Full incremental backup,

memoryefficientbackup yes

Incremental-by-date incremental backup

Journal-based incremental backup

© 2007 IBM Corporation41 Potpourri du Backup

Page 42: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Improved incremental backup memory usageImproved incremental backup memory usage

Incremental backup, 21 million files workload, Win2003 server, Win2003 x64 client, LAN, TCP/IP, Disk stgpoolIntel 4-way 3GHz (Paxville dual core, hyperthread) server, IBM x346, 2-way 3.4GHz client, 4GB, Compression No,

Encryption No Storage pool CRC No Bufpoolsize 131072 Txngroupmax 256 TXNBytelimit 2097152 NTFS filespace

DescriptionElapsed time (seconds)

0% changed data 5% changed data 20% changed data

Full incremental backup, memoryeff diskcachem, default db key size n/a 42496 45117

Encryption No, Storage pool CRC No, Bufpoolsize 131072, Txngroupmax 256, TXNBytelimit 2097152, NTFS filespace

Full incremental backup, memoryeff diskcachem, optimal db key size n/a 21327 24350

Full incremental backup, memoryefficient no 23652 27176 30088

Full incremental backup, memoryefficient yes 12724 16244 20714

Incremental-by-date incremental backup n/a 13681 17974

Journal-based incremental backup 2 12075 Failed

The disk cache method was faster than 'memoryefficientbackup no' method on Windows x64

Tivoli Storage Manager Version 5.4.0 Incremental Backup Performance

Windows 2003 x64 Client (3.4GHz, 4GB RAM)21 million 1KB files workload

64bit OS allows larger process virtual memory, but large real memory demand cause pagingThe disk cache method disk I/O is more efficient than paging disk I/O

25000

30000

35000

40000

45000

50000

ed ti

me

(sec

)

Full incremental backup,memoryefficientbackup diskcachemethod,default db key size

Full incremental backup,memoryefficientbackup diskcachemethod,optimal db key size

Full incremental backup,memoryefficientbackup no

0

5000

10000

15000

20000

0% 5% 20%

Percent changed dataEl

apse Full incremental backup,

memoryefficientbackup yes

Incremental-by-date incremental backup

Journal-based incremental backup

© 2007 IBM Corporation42 Potpourri du Backup

Page 43: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

AgendaAgenda

Backup techniques– What was available before ADSM/TSM came on the scene– TSM Progressive incremental backup model and its derivatives– Image backupImage backup

How to determine which backup technique best fits– What problem are you trying to solve– Windows– Other operating systems

© 2007 IBM Corporation43 Potpourri du Backup

Page 44: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

What Problem Are You Trying to Solve?What Problem Are You Trying to Solve?

RTO and RPO

Progressive Incremental backup is still most comprehensive backup– Best method to detect any type of file change– Individual file restore

$64,000 question is “Why can’t you use progressive incremental backup?”backup?– Memory– Backup window

© 2007 IBM Corporation44 Potpourri du Backup

Page 45: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

AgendaAgenda

Backup techniques– What was available before ADSM/TSM came on the scene– TSM Progressive incremental backup model and its derivatives– Image backupImage backup

How to determine which backup technique best fits– What problem are you trying to solve– Windows– Other operating systems

© 2007 IBM Corporation45 Potpourri du Backup

Page 46: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

toolstools

tsmstats.inits stats– Introduced in TSM 5.4.0 Backup-Archive Client

direx exedirex.exe– Coming soon from TSM

others

© 2007 IBM Corporation46 Potpourri du Backup

Page 47: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

tsmstats initsmstats.ini

[fileSystemStatistics.\\patmos\m$]

maxPath=68

totalLocalFiles=1381

totalLocalDirs=552totalLocalDirs=552

maxDirCount=30

maxDirName=M:\test\Hiragana

© 2007 IBM Corporation47 Potpourri du Backup

Page 48: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Windows – In Your ToolkitWindows In Your Toolkit

Progressive incremental backup

Journal based backup

Virtual mount points

Adaptive sub-file backup

Image backup (on-line and “used block only”)

© 2007 IBM Corporation48 Potpourri du Backup

Page 49: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Windows – Attacking Memory ProblemsWindows Attacking Memory Problems

You need to solve the memory problem first– JBB needs progressive incremental before the journal is enabled– JBB will fall-back to a progressive incremental at some point– Incremental by date needs to use a periodic full progressive

incremental at some point

Is memoryefficient yes going to workIs memoryefficient yes going to work– Run direx.exe to find out biggest directory bottleneck

• Most objects in single directory * 300 < 2 GB ?– Then use memoryefficient yesThen use memoryefficient yes

• Most objects in single directory * 300 > 2 GB ?– Memoryefficient yes not going to solve the problem

> Use memoryefficient diskcache or> Break-up file system orBreak up file system or> Use image incremental

If memoryefficient backup is going to help, you can now consider JBB to further reduce backup window

© 2007 IBM Corporation49 Potpourri du Backup

consider JBB to further reduce backup window

Page 50: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Windows – Attacking the Backup WindowWindows Attacking the Backup Window

Consider JBBCo s de J– JBB will have to fall back to progressive incremental model at

some point• Notification buffer overrun (this is a “safety valve”)

– How often is this going to happen in your environment

H ft t l t thi– How often can you tolerate this

© 2007 IBM Corporation50 Potpourri du Backup

Page 51: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

JBB – Tracking Down the Buffer Overrun Safety ValveJBB Tracking Down the Buffer Overrun Safety Valve

Run TSM journal daemon stand-alone (i.e., not u S jou a dae o sta d a o e ( e , otcommunicating with the B-A client)– Look for notification buffer overflow errors in error logg

New TSM 5.4.1.2 trace flag helps you see activity in journal daemonj– Can use to tune journal daemon exclude list

© 2007 IBM Corporation51 Potpourri du Backup

Page 52: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Creating Stand-Alone JBB DaemonCreating Stand Alone JBB Daemon

Modify journal .ini file to journal one or more file systems– Can use configuration wizard (just don’t start the service).

Modify the .ini file– Add trace semantics (see next page)– Modify communications (named pipe) so that the B-A client can’t talk to it

Start the journal serviceStart the journal service– Alternatively run in foreground: tsmjbbd.exe i

© 2007 IBM Corporation52 Potpourri du Backup

Page 53: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

tsmjbbd initsmjbbd.ini

…[JournalSettings]

fl jbbfilTraceflags=jbbfilemoncsvTracefile=M:\jbbtraceJournalPipe=\\.\pipe\bogusPipeE l jbb lErrorlog=jbberror.logJournalDir=C:\Program Files\Tivoli\TSM\baclient…

© 2007 IBM Corporation53 Potpourri du Backup

Page 54: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Buffer Overrun Safety Valve (jbberror log)Buffer Overrun Safety Valve (jbberror.log)

……06/04/2007 15:48:02 fifoQInsert(00A4ED04): Queue threshold reached, thread 960 returned from wait.06/04/2007 15:48:02 fifoQInsert(00A4ED04): Queue threshold reached, thread 960 will block until queue is under threshold.06/04/2007 15:48:02 fifoQInsert(00A4ED04): Queue threshold reached thread 960 returned06/04/2007 15:48:02 fifoQInsert(00A4ED04): Queue threshold reached, thread 960 returned from wait.06/04/2007 15:48:05 psFsMonitorThread(tid 960): Notification buffer overrun for monitored FS 'C:\', journal will be reset.06/04/2007 15:48:05 jnlDbCntrl(): Resetting journal for fs 'C:'.…

© 2007 IBM Corporation54 Potpourri du Backup

Page 55: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Now What?Now What?

How often are you seeing buffer overruns?o o te a e you see g bu e o e u s– More often then backup window?

• you will not be able to use JBB effectively– Sometimes?

• You have to decide if it is acceptableNe er?– Never?• WOOHOO!

JBB is a good fit• JBB is a good fit

Don’t forget you can tune JBB’s exclude list and eliminate unnecessary journal entries

© 2007 IBM Corporation55 Potpourri du Backup

eliminate unnecessary journal entries

Page 56: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Image BackupImage Backup

Consider using imageg g– Can’t resolve the memory problems

– Too many changes (> 30%) for JBB or progressive incremental backup

– Most of the file system is small files (avg. size < 1 MB)

– Need faster recovery time then file-level restore can provide

Try to keep the file change rate < 30% when using image and incremental combinations

© 2007 IBM Corporation56 Potpourri du Backup

Page 57: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

WindowsWindows

Other thoughtsg– Compression … if you have relatively new CPU and no hardware

compression… why not?

T k f l d li t– Take care of your exclude lists• Aggressive use of EXCLUDE.DIR where possible

– Open File Support• Are there files that are open that you need protected, e.g., ntuser.dat on

workstations?• … and can you not get the files off-line in some other manner, e.g., presched

command?command?

© 2007 IBM Corporation57 Potpourri du Backup

Page 58: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

AgendaAgenda

Backup techniques– What was available before ADSM/TSM came on the scene– TSM Progressive incremental backup model and its derivatives– Image backupImage backup

How to determine which backup technique best fits– What problem are you trying to solve– Windows– Other operating systems

© 2007 IBM Corporation58 Potpourri du Backup

Page 59: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

AIX – In Your ToolkitAIX In Your Toolkit

Progressive incremental backupProgressive incremental backup

Journal based backupJournal based backup

Virtual mount pointsVirtual mount points

Adaptive sub-file backupdapt e sub e bac up

Image backupg p

© 2007 IBM Corporation59 Potpourri du Backup

Page 60: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Linux – In Your ToolkitLinux In Your Toolkit

Progressive incremental backupg p

Journal based backupJournal based backup

Virtual mount pointsVirtual mount points

Ad ti b fil b kAdaptive sub-file backup

Image backup (on-line)

© 2007 IBM Corporation60 Potpourri du Backup

Page 61: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

UNIX – In Your ToolkitUNIX In Your Toolkit

Progressive incremental backupg p

Journal based backupJournal based backup

Virtual mount pointsVirtual mount points

Ad ti b fil b kAdaptive sub-file backup

Image backup (off-line)

© 2007 IBM Corporation61 Potpourri du Backup

Page 62: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Macintosh and NetWare – In Your ToolkitMacintosh and NetWare In Your Toolkit

Progressive incremental backupg p

Journal based backupJournal based backup

Virtual mount pointsVirtual mount points

Ad ti b fil b kAdaptive sub-file backup

Image backup

© 2007 IBM Corporation62 Potpourri du Backup

Page 63: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Other Platforms – Attacking Memory ProblemsOther Platforms Attacking Memory Problems

You need to solve the memory problem first– JBB (AIX) will have to be enabled; and it will fall back to a

progressive incremental model at some point– Incremental by date needs to use a periodic full progressive– Incremental by date needs to use a periodic full progressive

incremental at some point

Is memoryefficient yes going to work– Run direx.exe to find out biggest directory bottleneck

• Most objects in single directory * 300 < 2 GB ?– Then use memoryefficient yesMost objects in single director * 300 > 2 GB ?• Most objects in single directory * 300 > 2 GB ?– Memoryefficient yes not going to solve the problem

> Use memoryefficient diskcache or> Break-up files system or

U i i t l> Use image incremental

If memoryefficient backup is going to help, now consider JBB (AIX) to further reduce backup window

© 2007 IBM Corporation63 Potpourri du Backup

( )

Page 64: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

AIX – Attacking the Backup WindowAIX Attacking the Backup Window

Consider JBBCo s de J– JBB will have to fall back to progressive incremental model at

some point• Notification buffer overrun (this is a “safety valve”)

– How often this will happen in your environment

H ft t l t thi– How often can you tolerate this

© 2007 IBM Corporation64 Potpourri du Backup

Page 65: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

JBB – Tracking Down the Buffer Overrun Safety ValveJBB Tracking Down the Buffer Overrun Safety Valve

Run TSM journal daemon stand-alone (i.e., not u S jou a dae o sta d a o e ( e , otcommunicating with the B-A client)– Look for notification buffer overflow errors in error logg

New TSM 5.4.1.2 trace flag helps you see activity in journal daemonj– Can use to tune journal daemon exclude list

© 2007 IBM Corporation65 Potpourri du Backup

Page 66: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

AIX - Creating Stand-Alone JBB DaemonAIX Creating Stand Alone JBB Daemon

Modify journal .ini file to journal one or more file systemsod y jou a e to jou a o e o o e e syste s

Modify the .ini fileAdd t ti ( t )– Add trace semantics (see next page)

– Rename daemon so that the B-A client can’t talk to it

Start the journal daemon

© 2007 IBM Corporation66 Potpourri du Backup

Page 67: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

tsmjbbd initsmjbbd.ini

…[JournalSettings]

fl jbbfilTraceflags=jbbfilemoncsvTracefile=M:\jbbtraceErrorlog=jbberror.log…

© 2007 IBM Corporation67 Potpourri du Backup

Page 68: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Now What?Now What?

How often are you seeing buffer overruns?o o te a e you see g bu e o e u s– More often then backup window? You will not be able to use JBB

effectively

– Sometimes? You have to decide if it is acceptable

– Never? WOOHOO! JBB is a good fit

Don’t forget you can tune JBB’s exclude list and li i t j l t ieliminate unnecessary journal entries

© 2007 IBM Corporation68 Potpourri du Backup

Page 69: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Image Backup – non-WindowsImage Backup non Windows

Consider using image– Can’t resolve the memory problems– Too many changes (> 30%) for JBB or progressive incremental backup– File system is at least 60% fullFile system is at least 60% full

• UNIX and Linux paying for bandwidth by backing-up unused blocks– Where no on-line image support is available, you can unmount the file system– Most of the file system is small files (ave. < 1 mb) y ( )– Need faster recovery time then file-level restore can provide

Try to keep the file change rate < 30% when using image and incremental combinationscombinations

© 2007 IBM Corporation69 Potpourri du Backup

Page 70: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

Other PlatformsOther Platforms

Other thoughts– Compression … if you have relatively new CPU and no hardware

compression… why not?– Take care of your exclude lists

• Aggressive use of EXCLUDE.DIR where possible– Snapshot-based file backup or using snapshotroot option with 3rd party

snapshot provider• Are there files that are open that you need protected?• Are there files that are open that you need protected? • … and can you not get the files off-line in some other manner, e.g., presched

command

© 2007 IBM Corporation70 Potpourri du Backup

Page 71: Tivoli Storage, IBM Software Group

Tivoli Storage, IBM Software Group

ConclusionConclusion

Myriad of backup techniques in TSM and more to comey ad o bac up tec ques S a d o e to co e

One size does not fit all

Start with progressive incremental backup at it is still most comprehensive backup

Move to other progressive incremental or image techniques as the need arises

© 2007 IBM Corporation71 Potpourri du Backup