Business Resumption From An Application Perspective Presented By: Steven J. Skidmore IT Director Martin Clearwater & Bell LLP Thomas Huson Director, Professional

BBusiness Resumption From An Application Perspectiveusiness Resumption From An Application Perspective

Presented By:Presented By:Steven J. Skidmore Steven J. Skidmore IT Director IT Director Martin Clearwater & Bell LLPMartin Clearwater & Bell LLP

Thomas Huson Thomas Huson Director, Professional ServicesDirector, Professional ServicesAdvanced Legal Systems, Inc.Advanced Legal Systems, Inc.

Lawnet 2003Lawnet 2003Networking & EducationNetworking & EducationApplications TrackApplications TrackAugust 21, 2003 10:00amAugust 21, 2003 10:00am

Session OverviewSession Overview

The session is based on a worst case scenario The session is based on a worst case scenario Total devastation to computer room facilitiesTotal devastation to computer room facilities

Core components of sessionCore components of session

– Decision ProcessDecision Process– Build Build – RebuildRebuild

Disaster RecoveryDisaster Recovery

Disaster Recovery – The process of Disaster Recovery – The process of recovering from an event that has recovering from an event that has negatively affected the operation of an negatively affected the operation of an organizationorganization

– Disaster has occurredDisaster has occurred– Management declares disasterManagement declares disaster– Activate disaster recovery planActivate disaster recovery plan

Key Players Key Players

The FirmThe Firm– Litigation Firm Litigation Firm

Main Office Located Midtown ManhattanMain Office Located Midtown Manhattan Two Satellite OfficesTwo Satellite Offices Medical Malpractice DefenseMedical Malpractice Defense 80 Attorneys80 Attorneys

– IT DepartmentIT Department IT DirectorIT Director 2 Support Staff Employees2 Support Staff Employees

– Consulting TeamConsulting Team Sr. EngineerSr. Engineer

Case Study ObjectiveCase Study Objective

To find and implement a disaster To find and implement a disaster recovery solution that will restore the recovery solution that will restore the Firm’s critical business applications Firm’s critical business applications within 72 hours.within 72 hours.

The Decision ProcessThe Decision Process

Assessing Our RiskAssessing Our Risk

– Changed Attitudes –Everything has Changed Attitudes –Everything has changed since 9/11.changed since 9/11.

– Could not afford not to Could not afford not to Interruption to business can be costlyInterruption to business can be costly

– Responsibility to clientsResponsibility to clients– Our location puts us at a high riskOur location puts us at a high risk

United Nations, Israeli Consulate, United Nations, Israeli Consulate, Grand Central StationGrand Central Station

– Anticipate the worst case scenarioAnticipate the worst case scenario

Factors determining a DR solutionFactors determining a DR solution

– Cost FactorCost Factor Had a fixed budget to deal withHad a fixed budget to deal with

– Comdisco Recovery CenterComdisco Recovery Center– Work with Consulting team and keep Work with Consulting team and keep

project in house.project in house. Utilize satellite officeUtilize satellite office


Time FactorTime Factor– Identify critical business processesIdentify critical business processes– How fast did we need to be operationalHow fast did we need to be operational

Connection FactorConnection Factor– Who was going to access the recovery Who was going to access the recovery

system?system? Identify Tier levelsIdentify Tier levels

– Where are they going to access from?Where are they going to access from? OnsiteOnsite OffsiteOffsite

– How were they access the system?How were they access the system? CitrixCitrix


Application FactorApplication Factor– Essential Applications (0-72hrs) = Essential Applications (0-72hrs) = Disruption in Disruption in

service exceeding 72 hours would service exceeding 72 hours would jeopardize the operation of the Firmjeopardize the operation of the Firm

EliteElite– Time / BillingTime / Billing– Case Management / Calendar Docket SystemCase Management / Calendar Docket System

MSExhangeMSExhange– EmailEmail– CalendarCalendar– ContactsContacts

iManage / WordPerfectiManage / WordPerfect– Document ManagementDocument Management– Word ProcessingWord Processing


Recommended Applications (72hrs – 1wk) = Recommended Applications (72hrs – 1wk) = Disruption in service exceeding 1 week would Disruption in service exceeding 1 week would jeopardize the operation of the Firmjeopardize the operation of the Firm

– Microsoft Office SuiteMicrosoft Office Suite– SummationSummation– Record AccessRecord Access– CD-Rom ApplicationsCD-Rom Applications– Internet AccessInternet Access


Non-Essential Applications ( > 1 week)Non-Essential Applications ( > 1 week)

– Nice to have – but would not Nice to have – but would not jeopardize the operation of the Firm if jeopardize the operation of the Firm if they were missingthey were missing


Decision Process ResultDecision Process Result

Warm SiteWarm Site– Long Island Satellite OfficeLong Island Satellite Office– T1 ConnectionT1 Connection– Off Site Backup Tape Storage FacilityOff Site Backup Tape Storage Facility– Employees who live 15mins awayEmployees who live 15mins away

Designing a Recovery Designing a Recovery EnvironmentEnvironment

Application Functionality in the Live Application Functionality in the Live Environment Spans 9 servers in 2 DomainsEnvironment Spans 9 servers in 2 Domains Compress basic functionality into other Compress basic functionality into other hardware at minimal costhardware at minimal cost The Mission Critical apps the Firm The Mission Critical apps the Firm Identified not conducive to simple tape Identified not conducive to simple tape restore. Specifically:restore. Specifically:

– ExchangeExchange– SQLSQL– Elite’s InformixElite’s Informix

Challenges:

Building a Recovery Warm SiteBuilding a Recovery Warm Site

Communication Redundancy Communication Redundancy Data Replication SchemeData Replication Scheme Active Directory ConfigurationActive Directory Configuration Solid Hardware at Recovery SiteSolid Hardware at Recovery Site New Software InstallationsNew Software Installations

Solution


Communication configurationCommunication configuration– T-1 frame between sitesT-1 frame between sites– T-1 at recovery siteT-1 at recovery site– Redundant linkRedundant link– Recovery site has own scope of valid Recovery site has own scope of valid ip’s ip’s

Hardware configurationHardware configuration– Single high performance server to Single high performance server to house Majority of appshouse Majority of apps– Additional “light-weight” box for 2Additional “light-weight” box for 2ndnd domaindomain

Solution


Software ConfigurationSoftware Configuration– New install of mission critical software New install of mission critical software from factory mediafrom factory media

Server/Active Directory (Into existing forest)Server/Active Directory (Into existing forest) CitrixCitrix MS ExchangeMS Exchange SQL/iManageSQL/iManage RSASecureIDRSASecureID EliteElite ArcserveArcserve

–Installed, Tested – Services disabledInstalled, Tested – Services disabled

Solution:


ReplicationReplication– Arcserve copy utilityArcserve copy utility

Runs every 4 hoursRuns every 4 hours Copies over all files with archive bit flaggedCopies over all files with archive bit flagged Resets flagResets flag

Active Directory ConfigurationActive Directory Configuration– Disaster Recovery OUDisaster Recovery OU– Security Group “Recovery”Security Group “Recovery”

For Different application pointers, drive mappings For Different application pointers, drive mappings etc. etc.

Solution:

Rebuilding the EnterpriseRebuilding the Enterprise

AssumptionsAssumptions– Catastrophic EventCatastrophic Event– May not Have Access to the Most May not Have Access to the Most Recent Backup SetRecent Backup Set– Localized EventLocalized Event– We Utilize a Full Backup Strategy We Utilize a Full Backup Strategy

Rebuild ProcessRebuild Process– SharesShares– SQLSQL– Elite InformixElite Informix– MS ExchangeMS Exchange


Obtain Most Current Backup SetObtain Most Current Backup Set– May be 48 Hrs oldMay be 48 Hrs old

Begin Restore Process to Bring Recovery Begin Restore Process to Bring Recovery Environment OnlineEnvironment Online

– Restore SharesRestore Shares Merge TapeMerge Tape “ “Replace newer files only”Replace newer files only” (Should not actually lay down any data)(Should not actually lay down any data) AppsApps DataData DocsDocs HomeHome ShareShareStartMenusStartMenus


Restore SQL Database for iManageRestore SQL Database for iManage– Merge SQL Backup tapeMerge SQL Backup tape– Restore to . . .\SQLBACKUP directoryRestore to . . .\SQLBACKUP directory– Note Backup (BAK) fileNote Backup (BAK) file– Note Transaction log files Newer than backup fileNote Transaction log files Newer than backup file– Restore Database in SQL Enterprise ManagerRestore Database in SQL Enterprise Manager

“ “Restore database by device”Restore database by device” Select “Force restore over existing databaseSelect “Force restore over existing database Select “Leave Database nonoperational but able to Select “Leave Database nonoperational but able to restore logs”restore logs”

– Apply Transaction Log Files in SQL Enterprise ManagerApply Transaction Log Files in SQL Enterprise Manager Must be done individually if more than oneMust be done individually if more than one Select “Force restore over existing database”Select “Force restore over existing database” When all Log Files Applied Set Database to Online ModeWhen all Log Files Applied Set Database to Online Mode Modify directory pointers in Database (File server/Index Modify directory pointers in Database (File server/Index server)server) Change application pointers in iManageChange application pointers in iManage Start iManage ServicesStart iManage Services




Informix database for EliteInformix database for Elite– Restore from tape via command Restore from tape via command line in KORN Shellline in KORN Shell– Does Not Involve ARCServeDoes Not Involve ARCServe

Onstat – LOnstat – L Onstat - KOnstat - K Ontape – ROntape – R Onmode – MOnmode – M Onstat – L (To Verify Online Status)Onstat – L (To Verify Online Status)


Microsoft ExchangeMicrosoft Exchange– Merge TapeMerge Tape– Start Exchange ServicesStart Exchange Services– Modify User Accounts via ADSIModify User Accounts via ADSI

Launch ADSI Edit Application and CLEAR Launch ADSI Edit Application and CLEAR for Each User:for Each User:

– homeMDBhomeMDB– homeMTAhomeMTA– mailNicknamemailNickname– All other msExch propertiesAll other msExch properties




Microsoft Exchange (Cont. . .)Microsoft Exchange (Cont. . .)– Create New Mailbox in AD Users and Create New Mailbox in AD Users and computerscomputers– Send test mail to activate mailboxSend test mail to activate mailbox– Brick level restore to new mailboxesBrick level restore to new mailboxes– Move user account into disaster Move user account into disaster recovery OUrecovery OU– Add account to “recovery” groupAdd account to “recovery” group

ConclusionConclusion

Disaster Recovery SolutionDisaster Recovery Solution– DR solution needs to be tailored to the DR solution needs to be tailored to the

Firm’s size, resources and geographic Firm’s size, resources and geographic locationlocation

– Anticipate worst case scenarioAnticipate worst case scenario– Buy as much as you canBuy as much as you can– Documentation is keyDocumentation is key– Stay honestStay honest– UpdateUpdate

Documents

Business Resumption From An Application Perspective Presented By: Steven J. Skidmore IT Director Martin Clearwater & Bell LLP Thomas Huson Director, Professional