Module 4: Tips, Tuning and Troubleshooting

Module 4: Tips, Tuning and Troubleshooting

Learning Objectives

At the end of this module, you will understand:

Minimizing initial synchronization time

DataKeeper™ registry settings

The DataKeeper™ command line interface

User-reported issues and resolutions

Known restrictions

August 10 Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 2

DataKeeper™ Tips, Tuning and Troubleshooting

Local Initial Synchronization




Addresses issue of limited bandwidth

Much faster than WAN-based resynchronization

Not a panacea


DataKeeper™ Tips, Tuning and Troubleshooting:


Setup Final IP Configuration on Source and Target





Create Static Routes and Test via Ping



Local synch, then relocate target

Connect to all servers.

Create job and mirror(s) with appropriate compression setting(s).

Complete initial synchronization.

Shutdown target.

Delete route to target from source.




Local synch, then relocate target



Performance Optimization




Compression




Replication Bandwidth Throttle




Registry Settings





HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ExtMirr\Parameters

Parameter Default Value(s)

DontFlushAsynchQueue none drive letter(s) of replicated volume(s)

WriteQueueHighWater 9600 decimal

WriteQueueLowWater 50 decimal

MaxResyncPasses 200 decimal

Registry Settings


WAN Performance Optimization


HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ExtMirr\Parameters

Parameter Default (decimal) Recommended (Decimal)

PingInterval 3000 10000 (max = 40000)

MaxPingMisses 1 12

NetworkRetries 1 10

NetworkRetryDelay 5000 10000

NetWriteTimeout 10000 30000

ValidateTargetRetries 9 9

Registry Settings


WAN Performance Optimization

Registry Settings

Global Setting: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters

GlobalMaxTcpWindowSize (REG_DWORD) = Window Size

Tcp1323Opts (REG_DWORD) = 1

For NIC being used for replication: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters

\Interfaces\{GUID}

TcpWindowSize (REG_DWORD) = Window Size

Where: Window Size = Bandwidth * Latency



Command Line Interface (EMCMD)




Service Related

STOPSERVICE

GETSERVICEINFO

Job Related

CREATEJOB

DELETEJOB

UPDATEJOB

GETJOBINFO

GETJOBINFOFORVOL

Volume Related

LOCKVOLUME

UNLOCKVOLUME

GETVOLUMEINFO

GETMIRRORVOLINFO

GETEXTENDEDVOLUMEINFO

GETCOMPLETEVOLUMELIST

UPDATEVOLUMEINFO

ISPOTENTIALMIRRORVOL

RESTARTVOLUMEPIPE

GETVOLUMEDRVSTATE




Volume Related

GETCONFIGURATION

SETCONFIGURATION

GETSOURCEMIRROREDVOLUMES

GETTARGETMIRROREDVOLUMES

ISBREAKUSERREQUESTED

Mirror Related

CREATEMIRROR

RESYNCMIRROR

BREAKMIRROR

PAUSEMIRROR

CONTINUEMIRROR

Mirror Related

DELETEMIRROR

DELETELOCALMIRRORONLY

GETMIRRORTYPE

GETRESYNCSTATUS

READREGISTRY

CLEARSWITCHOVER

SWITCHOVERVOLUME



Command Line Interface

EMCMD (used by LifeKeeper)

Bitmap Related

GETREMOTEBITMAP

MERGETARGETBITMAP

Rewind Related

ENABLEREWIND

DISABLEREWIND

SUSPENDREWIND

GETREWINDENABLED

GETREWINDSUSPENDED

GETREWINDLOGFILE

REWINDTAG

FLUSHREWINDLOGFILE




Usage:

EMCMD <ip address> GETMIRRORVOLINFO <letter>

Command: emcmd 192.168.251.98 getmirrorvolinfo e

Output: E: 1 192.168.251.98 192.168.252.98 1

Parameter Description

E Mirrored volume

1 Current ROLE of the system specified 0 = None 1 = Source 2 = Target

192.168.251.98 SOURCE system

192.168.252.98 TARGET system

1 Current STATE of the mirror 0 – None 1 – Mirroring 2 – Resyncing 3 – Broken 4 – Paused 5 – Resync Pending




Use Case: Manual Switchover, Source Out of Service EMCMD <target ip> SWITCHOVERVOLUME <letter>

Use Case: Manual Switchover, Source Destroyed

EMCMD <target ip> DELETELOCALMIRRORONLY <letter>

EMCMD <target ip> CLEARSWITCHOVER <letter>

Source must be off-line, otherwise a mirror split-brain condition may be created.





Use Case: Target as Tape Backup Source

Tape backup tools may be used without application-aware agents

Scripted/automated

Description Command

Pause the mirror on

the source system

specifying IP

address of target

EMCMD <source ip> PAUSEMIRROR <letter> <target ip>

Unlock volume on

target system. EMCMD <target ip> UNLOCKVOLUME <letter>

Backup or copy

data from target

system.

copy; xcopy; ntbackup; backup; …




Description Command

Lock volume on

target system. EMCMD <target ip> LOCKVOLUME <letter>

Continue mirror on

source system EMCMD <source ip> CONTINUEMIRROR <letter> <target ip>

When mirror paused and unlocked, writes to source are flagged in bitmap. Sent to target when mirror continued (partial resynchronization).

When target unlocked, it is writable. Any blocks written on target overwritten by source blocks when mirror continued (partial resynchronization).

Use Case: Target as Tape Backup Source



Use Case: Disk-to-Disk Copy

Mirror Target

Mirror Source

Replication



Troubleshooting, Known Issues and Restrictions



“Split Brain” Scenario

Communications between systems lost

Target volume brought on-line

Communications between systems re-established

Event logs indicate there are two sources

Mirror paused on both servers



Split Brain



Split Brain Recovery

Determine which server should be source

Choose mirror source

Click “OK”


Mirror partially resynchronized

Disconnect/Reconnect GUI from/to servers if status incorrect

Source

Server

Mirror Source

Target

Server

Mirror Target

Resynch

Volume X: Volume X:


Issues and Resolutions

Component Message/Issue Cause Suggested

Action

User Interface Cannot connect to

servers

NetBIOS (137-139)

and/or SMB (445) ports

blocked by firewall

Open ports

User Interface

Can connect to servers,

create job and select

volumes for source and

target, but mirror

creation fails

DataKeeper dispatch

port (9999) or mirror

ports (10000-10025)

blocked by firewall

Open ports

User Interface No volumes available

for mirroring

Only C: available or

volume contains page

file

Create new volume or

remove page file from

volume





Action

User Interface

Failed to create mirror.

Target System event

log entry contains error

code: 0xC0000055

Distributed Link

Tracking Client service

enabled and active

Stop and disable

service on both source

and target

User Interface

Mirror created, but not

stored in job. Event log

entry contains “File:

.\GuiThread.cpp Line

3099 …”

Local security policy for

“Network Access: Let

Everyone permissions

apply to anonymous

users” is disabled.

Enable policy on all

servers.

System Event Log Create mirror failed.

vmms.exe program has

volume open

preventing DataKeeper

from obtaining volume

lock

Stop vmms.exe

program





Action

Windows Clustering

Newly created resource

appears off-line but is

unlocked

Expected behavior Bring resource on-line

Windows Clustering

DataKeeper Volume

resource not available for

use

DataKeeper installed

before cluster name

resource created

Perform repair install of

DataKeeper

Windows Clustering

DataKeeper Volume

resource not available for

use. Event log contains

“DataKeeperVolume.dll,

Error: 70”

All cluster nodes were

not on-line

Bring all cluster nodes

on-line



Known Issues and Workarounds

Component Message/Issue Cause Suggested Action

Performance Monitor

on Windows Server

2008 64-bit

No DataKeeper counter

logs available in

Performance Monitor

DataKeeper counters

not compatible with

64-bit Performance

Monitor

Use 32-bit Performance

Monitor (under

%WINDIR%\SysWow64\)

Performance Monitor

on Windows Server

2003

Counter logs do not

work on Windows

Server 2003

Known issue to be

resolved in future

release



Known Restrictions and Recommendations

DataKeeper home directory path name must be 32 characters or less

Pause mirror before running CHKDSK on source volume

Pause mirror before running defragmentation on source volume

Heavy write activity when using synchronous mirror negatively effects application performance

Resizing mirrored Dynamic Disks not supported

Create bitmap directory before relocating bitmap files

Bitmaps on Dynamic Disks that are not “System” are not supported



Summary



Summary

Minimizing initial synchronization time for WAN deployments

DataKeeper™ registry keys and settings

The DataKeeper™ command line interface and several use cases

How to recover from a “split-brain” situation

User configuration oversights and errors

Known restrictions and explanations


End Module 4

Lab: Tips, Tuning and Troubleshooting

Copyright © 2000-2008 by SteelEye Technology, Inc. All Rights Reserved. 36

Documents

Module 4: Tips, Tuning and Troubleshooting