NetWare5 and Advanced NDS Management - course designed for: Dynamic Mutual Funds, 40 King St. W. Toronto

NetWare5 and Advanced NDS Management

-course designed for:

Dynamic Mutual Funds, 40 King St. W.Toronto

Introductions

Jim Gillett Contract [email protected]– CNA 3,4,5 CNE 4,5,– CNI – MCP MCSE MCSB– MCT

Class 9:00 am – 4:00 pm

Course Outline

Day One- – Overview of NDS and eDirectory – Good Preventive Maintenance Measures for

the NDS Database– NDS Management Tools & Logic Source

Day Two-– Understanding How NDS Processes Management Tasks– The Relationship between NetWare Servers and NDS

Day Three-– NDS Troubleshooting

NDS and eDirectory

eDirectory (NDS)

Benefits (printout)Tree DesignPartitionsReplicas Synchronization

Good Preventive Maintenance Measures for

the NDS Database

Network Physician(?)

Basic NDS Health Check

– NDS version– Time Synchronization– Partition & Replica

Continuity

Complete NDS Checkup– Basic Checks plus -– NDS background

Processes:• External references• Obituaries• Remote server IDs• Unknown objects• NDS schema

– SET Parameters

Your NDS Tree- Static or Dynamic?

Static NDS Tree– Simple changes daily– New Partition or Server every

few months • MONTHLY CHECKUPS

Dynamic NDS Tree– Partition or Server weekly– Redesigning the Tree– Major upgrading

• WEEKLY CHECKUPS

Health Check ToolKit

NDS Manager (workstation-based)DSRepair (server-based)DS Diagnostics (server-based, reports)DS Trace (granular, view

background processes)

Use these tools to monitor, diagnose, repair and update NDS replicas and versions.– Each tool has advantages and useful features.

Additional useful tools

CRON command-line SchedulerServer logs and Records Server name,– NetWare revision NDS version IP,IPX numbers Partitions and replicas Hardware upgrades– Server abends, failures, error messages(*.err; *.log) Addition of software, patches, service packs eripheral devices

Exercises – 60 minutes

Exercise #1 Work with NDS ManagerCreate a report with DsDiag.Schedule an event with CRON.Work with DSRepairCreate several DS Trace

script files and observe results.

NDS Management Tools &Logic Source

NDS Error Guidelines

1st be sure physical connectivity exists between servers to allow synchronizationBe patient; allow time for NDS to synch after changesUse error codes and server documentationUsually NDS error conditions have occurred because of an inability of a synchronization process to complete.

Troubleshooting Method

Clearly define problemDefine possible causes (check server records,

error logs etc.)Assess possible solutionsBackup then try a solutionAllow time for effectDocument the solutionPlan how to avoid a repetition

Troubleshooting Tools

DSRepair identify and repairDSBrowse identify (NW5)NDS Manager identify and repairDSTrace identify and verifyDSView identify (NW4)Logic Source researchKnowledge Base research

DSRepair

Syntax >> DSREPAIR -[switch]– A Advanced mode– L Log file – U Unattended– RC Creates a database backup (dsrepair.dib)– P Flags unknown objects as reference– XK2 (killer switch 2) Arbitrarily removes all replicas

and turns all objects into ext. references– XK3 (killer switch 3) Further to XK2, clears the

‘backlinked’ status of all ext. references.

The DSRepair Log Files

Time Synchronization Log– Server name– DS version– Replica Depth– Time source, In Synch, Delta

Replica Synchronization Log– Partition Name and Replicas Stored on this Server– Time of last sync & Error Codes if any

DSRepair Advanced Options

Repair Local DatabaseServer InformationReplica & Partition Operations (many options)Check Volume ObjectsCheck External references (backlinks, obituaries)Schema and New EpochDatabase Dump file

DSView (NW4) & DSBrowse (NW5)

Allows detailed view of NDS objects and attributesWith DSBrowse, can delete objects or “receive” or “send” attributes or values of a specific object

(take time and try out both tools)

NDS Manager

Explain and/or Demonstrate – Partition continuity– Replica synchronization– Send updates– Receive updates– Check and/or Update DS version– Delete server– Remove server– Assign new master replica– Context-sensitive Help files

DS Trace

Flag Types * + - !Viewing Filters ( +, - )Force Actions ( * )Tunable Settings ( ! )Set TTF= ON/OFF Set DS Trace = ON/OFF– See DSTrace printout for all the detailed

switches

Research Tools

Knowledge Base– At support.novell.com– Manuals and TIDs

Logic Source for NDS– Advanced NDS Documentation– On Cdrom (check now)

Exercises

When you try to add a replica of the [Root] partition to a server, you get an error. (Exercise #2- 90 min.)Using troubleshooting method and NDS management and research tools, determine the cause and potential solutionsImplement the solutionDocument the problem and its resolution

End of Day OneSee you tomorrow!!You may leave when ready!


Dynamic Mutual Funds, 40 King St. W.

Toronto

Understanding How NDS Processes Management Tasks

Why Be Patient?

Because the Directory is a distributed (split up), and replicated (copied) database it takes some time for changes to be updated to all copies or replicas of the database.The term ‘loosely consistent’ can properly be applied to the NDS Directory.

Types of Changes

Simple– Normal day-to-day

administration of the Directory tree.

• Creating leaf objects• Modifying attributes of

leaf objects

– Can be performed at any time, multiple

locations

Complex– Higher level

administration tasks• Partitioning • Adding replicas to server• Merging partitions• Redesigning tree• Moving containers (Ous)

– Be sure one operation is completely synchronized before starting another

Handling NDS Wisely

Some Guidelines:– Do health check before any complex operation– Control complex NDS management tasks centrally– Perform complex tasks during off-hours if possible – Backup database using DSRepair –RC command– Allow time for the replicas to synchronize normally – Verify that one operation or part of an operation fully

completes before proceeding to another

Partitions

Creating Partitions, Adding Replicas

Be sure that the servers holding the master replicas needed to perform this operation are available. (If a server receives a replica of a certain partition it will also receive a ‘subordinate reference’ to any child partitions).Ensuring that all needed servers are on-line will allow the operation to complete promptly.Allow time for replica lists to be updated and any changes to be propagated to all servers holding replicas of the updated partitions.

DemonstrationRun DSRepair for 0 errors then:– create a new partition – Create a child partition of the new partition– Unload NDS from a server holding a replica of the parent partition

(unload DS.nlm)– Merge child partition with parent

Set DSTrace=on and Set DSTrace=*H– Observe Directory screen display

Run DSRepair and notice any errorsThe point? Simply that you need to allow one NDS operation to completely finish (synchronize) before attempting another.

Looking Behind the Scenes

DS Diagnostics allows you to generate NDS status reports– Load DSDIAG.nlm– Select preferences– Select generate report– Select parameters of report– Generate desired report and examine

Looking Behind the Scenes (II)DSTrace with –ST switch creates more detailed statistics on screen-Some useful switches>>

What are the Background NDS Synchronization Processes?

Replica synchronization JanitorFlat cleanerPurgerObituaryLimber

Replica Synchronization

Maintains consistency of data between replicas3 levels– immediate (10 secs hold time, most events)– Slow (30 secs hold time, attribute modifications, etc.)– Heartbeat (or skulker) every 30 minutes, automatic, default– To display

• Set dstrace=on• Set dstrace=+s (or, +skulker, or +sync)• Set dstrace=+misc• Set dstrace=+part• Set dstrace=*H (to force sync process)

Janitor process

Makes sure deleted objects are purged from each replica (using the obituary and flat cleaner processes)Verifies information re: replicas on server, status of server and time synchronization.Periodically optimizes databaseRuns on database initialization and again every 2 minutes

Flat cleaner

Scheduled by janitor to occur every 60 minutesPurges unused bindery (NW3) and external reference objects and attributesRemoves obituaries (deleted objects) if in purgeable stateTo observe:

• Set dstrace=+J• Set dstrace=+misc• Set dstrace=*F

Purger Process

Purges unused objects and attributes Processes obituaries through their different statesTriggered by synch processTo observe:

• Set dstrace=+J• Set dstrace=*H

Obituary Process

4 step process of deleting objects– 1 (Flag=0000) Object deleted- initially just on local

replica, others not yet notified– 2 (Flag=0001) Notified- all replicas informed of

change– 3 (Flag=0002) OK to purge- all replicas respond,

change implemented on their side– 4 (Flag=0004) Purgeable- really gone

Limber Process

Verifies server credentials such as-– Network address– NetWare version– Distinguished names of servers on Replica List – Name of the Tree– Authentication of credentials

To observe-– Set dstrace=+Limber– Set dstrace=*L

Schema Synchronization

At initialization, then every 240 minutesEnsures updates, additions to schema are sent to all replicasTo observe-– Set dstrace=+schema– Set dstrace=*SS

Backlinker ProcessTakes care of external referencesExternal reference are quick pointers to servers holding replicas of a certain object, stored on a server that has referenced that object but does not have a replica containing that object itself.They are not needed if the server later receives a replica containing that object, or if partitioning changes make it invalid.The Backlinker process runs 2 hours after initialization and every 780 minutes after.To observe-Set dstrace=+blink (or +backlink)

-Set dstrace=*B

NDS database is initialized when-

Volume SYS: is mountedDS.nlm is unloaded and reloadedIs forced by typing – Set dstrace=*.

To view completion type– Set dstrace=+init– Set dstrace=+misc– Set dstrace=+J

Exercises

Do exercises 3 and 4 (printouts, 60 minutes) Use DSTrace to view & confirm results. Be sure all obituaries are resolved. Using NDS Manager– View partitions– Create partitions– Merge partitions– Move containers

Types of Replicas

Master– First, needed for partitioning operations

Read/Write– Additional replica(s), fault tolerance

Read/Only– Cannot directly make changes or provide authentication

Subordinate Reference– Created by system, holds replica list or ring

External References

Basically a very simple pointer or placeholder to an object that is referenced by a server for which there is no replica on the server.An external reference just holds a few attributes of the object.Its purpose is to cache references to external objects for quicker access.A backlink is an attribute of an external reference which points to the location of the real replica holding the object and all its attributes.

Partition & Replica guidelines

Do not create unnecessary partitionsPartition at upper layers based on location Maintain fewer than 3500 objects per partitionFor fault tolerance have at least 3 replicas of each partition (esp. [ROOT] partition)Partition for bindery services if applicable (NW3)Use WAN Traffic Manager if applicable to off-load synchronization to low-use periods

NetWare5- Faster Synchronization

NetWare4 uses Sync-up-to Vector or timestamp– Replicas communicate and

synchronize one at a time,around the ring

– Read-only replicas and subordinate references can trigger synch process

– Replicas are searched sequentially for changes

– One object per synch packet.

NetWare5 uses Transitive Vector– Each server contains a list of

timestamps of all replicas in the ring, not just those on the one server.

– Read-only replicas and subordinate references do not trigger synch process

– Changes are cached for each replica, no sequential search

– Multiple objects per packet.

Replica States & Exercises

see replica states printout (support.novell.com)Exercises (30 min.)-– Add a replica to a server– Delete a replica– Change a replica type– Receive updates (local replica overwritten; cannot

perform on a master replica)– Send updates (local changes sent)

The Relationship Between NetWare Servers and NDS

Removing a Server from the Tree

Holiday Group Tour– where’s Mary?? Problem!– She’s got our funds! She knows schedule!

What does server do?– Does it hold master replica(s)? Move them.– Is it a time source server? Assign another.– Important apps? (GroupWise) Move them.

Inform group before removing. – (Remove NDS)

Is the removal temporary? – Use placeholders.(Nwconfig-NW5; Install-NW4)

How to Remove a Server

1. Run DSRepair2. Remove Master replicas from server3. Change if time source server4. If removal is temporary use a placeholder.5. Remove NDS using NWConfig.6. Verify all references removed.Can use ‘DSRepair –dsremove’ if necessary.

The –dsremove switch

‘NWConfig –dsremove’ allows arbitrary removal of NDS from a server (caution!)– Does not verify replica exists– Does not synchronize first– Does not stop on any errors– Does not require admin log-in– Might not remove all references

as cleanly as normal method.

– Exercise 6 – 30 minutes

Changing Server Credentials

Can change server name and ID.Server ID (formerly internal ipx #) random 8-digit hex number- may want to standardizeChange one at a time, restart serverCorrect volume names and license assignmentsLimber process verifies server IDsUse DSTrace ON, +limber, *L, *H to observe results ***Exercise #7, 30 minutes***

The Limber Process

Upgrading Server HardwareBackup the server filesRun DSMaint, Install(NW4) or NWConfig(NW5)– First record Server name and ID– Select “Save local DS information prior to hardware upgrade”

or “Prepare NDS for hardware upgrade”. NDS is now locked.– Do not perform any other replica or server changes during this time.– A file will be created called either Backup.ds or Backup.nds.– Copy this backup file from sys:\system to a workstation.– Upgrade the hard drive, with same version and server packs, into a

temporary tree. Log in from a workstation.– Copy the backup.nds file to sys:\system then remove NDS and restore

from the backup file using DSMaint, Install, or NWConfig

***Exercise #8 (30 minutes)***

When It Crashes

If a hard drive crashes-– Hard drive failure, overheating,

power supply or other hardware failure.

Documentation needed– Server name, ID, replicas, services etc.

Can use ‘DSMaint –PSE’ switch– Premium support engineer– Place server references on a

placeholder object– ‘Delete’ crashed server and volumes– Replace failed server– Use DSMaint to restore server

references, tape for file system.

Placeholder


Backups and RestoresFirst-line defence-– Partition replicas are an ‘on-line’ dynamic backup– Maintain at least 3 of each partition

Tape Backups-– the NDS Directory, File System and Server-specific Info(5 files)

• (5 files- servdata.nds,dsmisc.log,volsinfo.txt,startup.ncf,autoexec.ncf)– Partial restore or entire tree – Restore NDS before file system so rights can be restored.

TCOPY (tcopy2.exe) and TBACKUP (tback3.exe) useful toolsDatabase Dump-– Run DSRepair –RC (Use CRON)– Creates sys:system\dsrepair.dib, must run on each server.

Backup Guidelines

Use Novell-certified backup hardware & softwareUse latest versions of TSA’s, drivers, etc.Use latest service packs, patches.Document your network diligently.Back up regularly and test the integrity of backups.Always restore NDS before the file system.If re-install operating system, re-apply service packs, patches etc. before restoring NDS and file system.


End of Day TwoOne more to go!!You may leave when ready!



Toronto

NDS Troubleshooting

Uncommon Issues

Inconsistent leaf objects

Time un-synched

Server 54 where are you?

Who are you and how did you get here?

Schema mis-matches

Dead but not gone (obituaries)

Looking closer…….

DSView- NW4.11, read-only, run on each serverDSBrowse- NW5, can delete, send or receive updatesNetWare Administrator- easy to read, but how to be sure reading from server interested in checking?? Unload ds.nlm from other close servers, as NWAdmin uses info from first server to reply. Check number of subordinates in each container, make

notes, then check the view from another server and compare total number of objects.

Restoring consistency-

‘Receive updates’ on problem server– Replaces replica on

server with over-write from another server containing a trusted replica

‘Send updates’ to other servers in ‘ring’– Non-destructive– Updates all the other

replicas


Time Synchronization

Why needed.What makes a timestamp?Time and IPXTime and IPMixed IP/IPXResolving time synchronization issues

(timesync) Why needed? (1)

Used to accurately order changes in file system.Used to accurately record message timesUsed by applications to record events.Especially used by NDS to maintain consistency

of updates across multiple replicas. The order of modifications must be accurately recorded so changes made on one object

in two replicas can be prioritized correctly.

(timesync)What makes a timestamp?

3 parts-– 1. UTC or Universal Time Coordinated (GMT)

– 2. Replica number where event took place

– 3. Event ID- a consecutive number from 1 to 65,535to keep track of multiple events in the same second.

(timesync) Time and IPX

Timesync is an easy yet comprehensive Novell proprietary time synchronization service– With NW4 it used IPX– With NW5 it can use either IP or IPX or both.– With IPX, there were 4 types of time servers

• Single reference-default authoritative time source• Secondary- default all but 1st server, accept time corrections• Reference- form ‘committee’ with 2 to 7 Primary time servers• Primary- communicate with Reference servers and vote on time

– Uses timesync.cfg, simple text file for set parameters• SAP, configured lists,directory tree mode, etc.

(timesync) Time and IPStill uses timesync.cfgSet timesources to NTP Internet sourcesSpecify and use standard NTP port 123Load monitor>server parameters-or-edit the timesync.cfg directlySet timesync configured sources=ON Set timesync time sources= <ip addr. of Internet time server>:123Single reference, Reference and Primary Servers all get their time directly from internet sources. If the connection fails to the internet they will use their own clocks temporarily.Pure IPX servers must be secondary time servers and require IP/IPX compatibility mode or IP and IPX on one server.

Time Un-synched?

If time is set backwards on an authoritative time server, NDS realizes a problem and declares a state called “synthetic time”Uses timestamp IDs and allows timestamps to remain accurate relative to each other but compresses events compared to real-time until NDS time converges with real time. (automatic process) Displays “synthetic time” warnings at the server console.

Resolving “Synthetic Time”

1. Do Nothing- (preferred)– If time reversal was not too

long, just wait. NDS will eventually ‘catch up’ to itself and the messages will stop.

2. Declare a ‘New Epoch’– Load DSRepair –A– Perform at off-peak hours– Choose advanced options>

replica and partition operations> repair

time stamps and declare a new epoch

Don’t Filter Out NTP port 123

Your firewall must permit UDP ports 123 and 524 for NTP and timesync packets out; also all UDP in (unless firewall supports dynamic filtering) for timesync packets and port 123 in for NTP across the firewall.

**Exercise #12 (30 mins)**

Server 54, Where Are You?-625 error –transport failureCheck physical connectionsUse ping / ipxpingCheck router logs (dropped pkts)Don’t span a slow WANCheck DS.nlm loaded on each serverDon’t filter ports- SAP 278, 26b; all RIP; TCP/UDP 524;UDP 123; TCP/UDP 427; TCP 2302;UDP2645They are used by- NTP, SAP, SLP, RIP,CMD, NCPServer IDs and public key can become corrupted –632 error:system failure-add / repair replica or use DSRepair –XK3 to re-backlink ext references. ***Exercise #13 (30 minutes)***

Who are you and how did you get here?

All objects have several mandatory attributes or properties.If a mandatory attribute of an object becomes corrupt, has no value, or cannot be located, the object becomes ‘unknown’.When product add-ons or upgrades are installed, the ‘schema’ of NDS may be extended and ‘viewing snap-ins’ added to allow a wider range of objects & properties.

How to get rid of unknown objectsBe sure NetWare Administrator has the required .dll’s (snap-ins) so it can ‘see’ the objects.If you are sure the unknown objects no longer exist (ex.-volume objects from a deleted server) simply delete them.If a leaf object can be easily re-created there is also no harm in deleting it.Using NDSManager or DSRepair, run the ‘Receive update’ option on the corrupted replica, or ‘Send updates’ from an uncorrupted replica. Run DSRepair –P which will flag all unknown objects as new objects. Their attributes will then be automatically updated the next time the replica ring synchronizes.

How to get rid of renamed objects

If replica A1 can’t communicate with replica A2 for a period of time, it is possible that 2 different objects could accidentally receive the same name.When communication is restored, NDS will see the clash and rename one object with a no._no. designation (ex.- 1_2)Solution- Be sure all replicas are now ‘ON’ and communicating then either delete or rename one of the objects. (***Exercise 14 – 30 mins.***)

Schema mis-matches

The schema is the layout of NDS, what objects and properties are allowed and can be understood.Every time you upgrade NetWare, or add a program like ZENworks, the schema is extended.All servers in the tree hold a copy of the ‘schema’ pattern and if the schema is extended this update is passed to all servers .This process occurs 15 seconds after initialization and then every 240 minutes after.For manual schema updates, DSRepair –A then Advanced Options>>Global schema operations. (***Exercise #15 –30 mins ***)


When an object is-– Deleted– Renamed– Moved

The actual purging of the object only occurs after-– Master replica says ‘Dead!’ (Flag 0000)– All servers notified (Flag 0001)– Servers report OK to purge (Flag 0002)– Entry actually purgeable (Flag 0004)


There are various types of obituaries based on the type of operation that was performed on the object(s).If obituaries get ‘stalled’ you may need to clean them up. How? >>


If an obituary is not cleanly removed, determine which server has not acknowledged the process by running dsrepair –A and looking at the log reports.– Find the object that was deleted but not OK’d to purge– Find where it was moved from– Find which server has not responded to the obituary process– Clear up the problem with that server.

Options to clear up the problem-– Restore the offending server’s communications with network– If server no longer exists, delete using NDSManager– Run DSRepair with the –XK3 switch on the offending server

The XK3 switch

Resets all external references on a server to cause the properties to be refreshed by the backlink process.In some cases will clear up stuck obituaries by allowing the process to go beyond Flag 0002.Note: If an offending obituary does not respond to these steps, contact Novell Technical Support for further assistance to solve the problem. ***Exercise #16 (30 min.)***

Practice Exercises

As time permits-– Exercises 17 to 26

End of Day ThreeOK all you NDS Experts!You may leave when ready!Thank-you for your

contribution!



Toronto

Documents

NetWare5 and Advanced NDS Management - course designed for: Dynamic Mutual Funds, 40 King St. W. Toronto