BP103 ”Top Chefs” Share Recipes for Avoiding Everyday Server Disasters

Marie Scott, Virginia Commonwealth UniversityFranziska Tanner, MartinScott Consulting(Gabriella Davis, The Turtle Partnership)

Legal● This slide presentation may contain the following copyrighted,

trademarked, and/or restricted terms:▬ IBM® Lotus® Domino®, IBM® Lotus® Notes®, IBM Lotus Symphony®, LotusScript®

▬ Microsoft® Windows®, Microsoft Excel®, Microsoft Office®

▬ Linux®, Java®, Adobe® Acrobat®, Adobe Flash®

Speaker introduction

Replication

Server Tuning

Securing Your Servers

Clustering

Mail Routing

Managing Clients

Wrap-up and Q&A

Agenda

4

Speaker Introductions• Marie Scott, Virginia Commonwealth University• Franziska Tanner, MartinScott Consulting• (Gabriella Davis, The Turtle Partnership)

● Combined 27 years experience working with Notes and Domino● Versions 4 – 8● 10 – 100,000 user sites● Combined 38 certifications across Domino, Websphere and

Workplace products

Speaker introduction

Replication

Server Tuning

Securing Your Servers

Clustering

Mail Routing

Managing Clients

Wrap-up and Q&A

Agenda

Disaster: Replication Stew● Users are reporting missing information in applications on some

servers

● You are not sure which databases are replicating reliably or if any missed their schedule

● There is a need to be alerted of missed replication schedules and cycles without having to manually check logs

How you got here

● Your Domino environment was inherited and “everything seemed to work fine so far”

▬ Nobody bothered reviewing the topology nor why system databases are not being replicated

● The ACL of some applications is preventing replication from happening, even though the server dutifully executes the connection document

▬ Replicating it both ways, every 5 minutes even, gladly wasting server resources

● Document count on applications is not the same across all servers

Recipe for Success● Understand your current replication model

▬ Mapping this out visually goes a long way and also helps you find flaws▬ Don't waste your time with the built in topology maps task

▬ Note one way, two way and specific databases you are in or excluding too

● Make sure all spokes get a turn▬ Don't get caught in a situation where all servers in Zumwald haven't replicated

in weeks▬ See earlier disaster where old data replicates back into the

environment▬ Stagger replication to where all servers don't go into overdrive at the top of

each hour

Recipe for Success● Validate your replication strategy

▬ Make sure you're not one-way replicating Admin4.nsf, names.nsf, certlog.nsf or events4.nsf if you want data to propagate normally

● Use scheduled replication probes to find out if any apps have missed their schedules

▬ Different probes can be created for individual applications of various replication intervals

▬ Replication error probes can tell you if the server encountered issues

Chef's Tips● Make templates and other infrequently changing applications low

priority by going to “Replication – Options for this Application” and setting the replication priority

▬ You can then make priority based replication connection documents

Chef's Tips

● If you have the resources, you can run more than one replicator task on your server – replicators=x

▬ Make sure you have enough resources before making changes

● Use any tools available to you to compare document count▬ There is a free one at http://bit.ly/8CZMTd

● Consider server utilization when assigning pull, push, pull-pull or pull-push replication to connection documents

▬ Pull-push makes the calling server do all the work but doing so utilizes the fastest method or server to server replication

12

Disaster: Garbage Pizza

● User complaining of no longer having access to applications

● Replication isn’t happening between some servers

● You notice some people who were terminated a month ago are back in the Directory

How you got here● Someone with Editor or higher access and an outdated replica has

replicated deletions back into the environment● Previously deleted person, program, configuration, group and server

documents made their way back into the Domino Directory● Anywhere groups are used to grant access is no longer working reliably

▬ ACL▬ Server connection documents▬ Admin access to servers▬ Readers fields in applications

Basic Replication Ingredients● Connection/replication documents

▬ Or manual replication● Domino Authentication● Matching Replica ID’s

▬ File name or folder location does NOT matter● ACL rights● Replication formulas● Field level rights● Replication history

▬ By default, Domino replicates documents that have changed since last time the application replicated

●

Recipe for Success● Anyone with Editor rights or above and

an outdated local replica is suspect

▬ Roles are ignored at this level▬ Minimize all Editor rights or above

● Scan “User Detail” in application properties for unusual activity

Chef's Tips● Try to avoid manual replication

▬ Circumvents replication formulas and pull/push typesettings listed in connection docs

● Don’t have more than one of any replica ID on servers▬ Check the “by replica id” view in your catalog.nsf

● Use “enforce consistent ACL” on apps if possible▬ Minimizes changes of erroneous ACL changes

● Setup monitoring events to alter you of missed replication cycles●

● Prevent local replicas from being created▬ Use 3rd party apps to “delete if not replicated in x days”

● “Ban” test servers to their own environment to avoid in-frequent server to server replication from causing this

Speaker introduction

Replication

Server Tuning

Securing Your Servers

Clustering

Mail Routing

Managing Clients

Wrap-up and Q&A

Agenda

18

Disaster: Transaction Log Jam● You see the following message on the server console:

▬ “Recovery Manager: Log File is Full try again later.”

● Mail items with large attachments are causing semaphore locks on mail.box and delays in mail delivery.

19

How You Got Here Transaction log files filed up the allocated space.

Confirm the configuration of the backup software that is to archive the transaction logs or the allocation of space for circular transaction logs.

Transaction logging was not disabled on log.nsf or mail.boxes (see advanced settings on database properties tab)

20

Chef's Tips Transaction logs should be located on a physical separate drive Use the following Notes.ini parameters to troubleshoot transaction log

deadlocks: Debug_Show_Server_Timeout=1 Debug_Capture_Timeout=1

Useful Notes.ini settings for transaction logging: MAILBOXDISABLETXNLOGGING=1 SCHEDULE_DISABLETXNLOGGING=1 RM_NO_LOG_OBJECTS_IN_MAILBOX=1 NSF_DONT_LOG_MAILBOX_BODY=1

LOG_DISABLETXNLOGGING=1

'Show stat database.rm.*' Confirm that backup software is certified for transaction logging.

21

Disaster: Burnt Database Souffle

● Databases are slower and slower to open.● Compact and other maintenance tasks are not finishing in time● Views are not refreshing without manual intervention.● Full-text indexes have not rebuilt.● Consistency checks are running on all databases after server crashes.

22

How you got here● All mail databases are on one mail path.● View indexes and full text indexes are on the same physical drive.● Transaction logs are not enabled.● Database storage system has not been optimized for Domino.● Physical connection to storage is over one controller.

23

Chef's Tips● Put mail files or other databases across multiple database paths

(directory links).

● Run transaction logs for faster startups (even if only using circular).

● Distribute physical drives across multiple disk controllers.

● RAID 0+1 or RAID 1E and use hardware RAID not software RAID.

● Use larger block size and matching stripe size

● Put view rebuilds on separate drive.▬ View_Rebuild_Dir=<directory path>

● 'show stat platform.log*'

● 'show stat database'

24

Disaster: Deep Fried Name Look-ups ● Name look-ups are taking longer and longer.● Web authentication hangs after userid/password are entered.● Servers stop responding intermittently for no apparent reason and then

resume normal service.● Individual documents in the names.nsf are delayed in opening.

25

How you got here● Additional views have been added to names.nsf with many columns

(requiring a refresh of each column).● Directory indexer task hasn't run on the names.nsf for several days.● Design refresh is running on names.nsf during production hours.● Mix-match on design elements (forms or views were older version).

26

Chef's Tips● Minimize the number of customized views in names.nsf. ● Store unused customized views in another database.● Perform routine maintenance on names.nsf (offline).● When upgrading the design of the names.nsf, make a master copy and

physically deploy a copy to servers (if possible) rather than replicate.

27

More Gadgets for your Notes.ini● Schedule_No_Validate = 1

● Server_Show_Performance=1

● Console_Log_Enabled=1

● Console_Log_Max_Kybtes=size (kb)

● Schedule_No_Calcstats=1

● Schedule_No_Validate=1

● Converter_Log_level=10

28

Top Chef Special: Activity Trends Activity Trends (Admin Client Server – Statistics)

Identify Trends or activity within 24 period

Useful for identifying critical issues (e.g., users using all server resources, runaway replication, too many transactions on names.nsf, etc.)

Needs to be setup and configured before you need information

29

Activity Trends

30

Chef's Tips: “Too much or too little salt”● Performance tuning vs. performance troubleshooting● Document every change and make only one change at a time.

▬ Documentation database or rem statements in Notes in with dates.▬ Run DDM probes to compare server setting periodically

● Review OS, network, disk storage system, and Domino performance statistics before making drastic modifications.

● 'show stat'● Server_show_performance=1

Speaker introduction

Replication

Server Tuning

Securing Your Servers

Clustering

Mail Routing

Managing Clients

Wrap-up and Q&A

Agenda

Disaster: Security Frappuchino• Your inherited your environment and suspect that unauthorized

people are gaining access to your servers

• You don't know which servers are being accessed or what setting to change to prevent this

• You want to find out if any server setting is changed as soon as possible

How you got here• You configured the servers correctly when they were installed but

over time, settings change and new features get added- over time changes are made in isolation to fix problem but

they can create their own security holes

- new features / settings are introduced with each version and often these settings still have their default value

-

• You don’t have management of every server in your organisation and are worried someone else could create a security hole

- Having Anonymous Notes access allowed on another server, where your “secure” application is replicated to

- Running HTTP on another server where your “secure” application is replicated to

- Running apps on a server allowing “more name variation with lower security” internet passwords which doesn’t enforce HTTP lockout for brute force password attacks

- Setting is on your server document – security tab

Recipe for Success: Ingredients● Use Domino Domain Monitoring (DDM) Best Practices To Check Your

Server Configuration▬ Security Best Practices Probe▬ Web Best Practices Probe▬ You would run these probes intermittently, such as monthly or after an upgrade

● Use DDM Best Practices To Check Your Server Configuration against a known ‘Good’ server that you want to use as a baseline

▬ Security Configuration Probe▬ Web Configuration Probe▬ Depending on the level of shared administration and risk in your organisation

you would run these probes either in real time, daily or weekly.

Why this is tasty● The “Best Practice” probes analyze each selected server according to

IBM shipped Best Practice criteria and make recommendations for security or performance changes

▬ The detailed reports provided enable you to validate your configuration▬

● The “Configuration” probes analyze each specified server’s settings against a ‘Guideline’ server you designate, generating a report of any mismatches

▬ The subsequent reports show if any server setting has varied from what you consider a standard configuration

▬ The probe can be set to review a very limited set of settings that you consider critical

▬ You can use different guideline servers for different types of Domino servers such as mail servers, application servers, web servers etc

Why this is tasty• The Database ACL Security probe will report into the DDM.NSF on

each server

• Those results will show as security problems for any databases with ACL's that exceed the rules you have set up

• Exceptions requiring higher access levels but you can review those then exclude them from further probes

• The Event Generator combined with an Event Handler will notify you of ACL changes in critical databases in real time so you can secure the database

Getting The Recipe Wrong• Events4.nsf needs to be replicated on every server in your domain.

Verify that each server within the domain should have the same events4.nsf replica id

• DDM.NSF has a replication selection formula by default that limits the contents of that db to its own server (designate a DDM collection server).

• The probes and DDM.NSF itself have no notification models, only Events and Statistics have those (create your own event generator).

Speaker introduction

Replication

Server Tuning

Securing Your Servers

Clustering

Mail Routing

Managing Clients

Wrap-up and Q&A

Agenda

39

Disaster: Clusters – Nuts or Not?● After a server crash of a clustered server, users report that they're

missing documents and unread marks.

● Database counts are not in synch between clustered servers; where are the missing replicas?

● One server in a cluster pair seems to be replicating changes to its cluster buddy, but the other server is not pushing changes.

● A user attempts to connect to their home mail database and is failed over to the cluster buddy even though their home server is up and running.

40

How you got here● Mismatched database directories, ACL, out–service-databases,

notes.ini parms, or connection documents are prime candidates for causing issues with cluster replicas.

● Remember to consider changes “in pairs” as a change on one cluster buddy, and not another may affect performance.

● Confirm that database properties are the same for each replica. If databases are moved be sure that they are moved to the same directory on the cluster buddy server.

● Do a health check on the cldbdir.nsf of each cluster server – this file can become corrupt.

41

Recipe for Success● Confirm the following notes.ini settings:

▬ SERVER_CLUSTER_DEFAULT_PORT=<clusterport>

▬ Server_Cluster_Probe_Port=<clusterport>

▬ Where <clusterport> matches the port name you assigned in the Notes.ini to be used for cluster traffic.

● Review CLDBDIR.NSF to see if files are being clustered and if replication is enabled.

● Check the number of Cluster Replicator tasks and Replica tasks that are running and included in the ServerTask list.

● In the files view of your Administrator client, check to see that ODS versions match as well as template versions match between replicas.

● Confirm that cluster servers are in the same Notes Named Network.

42

Chefs Gadget: Cluster Analysis Tool● Cluster Analysis (Admin Client * Server – Analysis – Analyze Cluster)● Can identify databases that have disabled replication, mismatched

ACL, missing replicas, and mismatched replication formulas.

43

Chefs Gadget: “Show Cluster” Console Command

● Show Cluster (Server Console)▬ Displays settings for the cluster – especially cluster port in use.▬ A quick ‘temperature’ check to assess cluster performance

44

Chef's Tips● Run Cluster Analysis at least once a month, especially after server or

database moves – to identify database replica orphans.

● After database moves on clusters, clean up full text indices to save disk space (the index files will not move with the replicas – they stay in place).

● Set up a separate private subnet for cluster replication if your cluster servers support mail databases or databases with lots of transactions.

● Schedule program docs to run at different time on each cluster (e.g., compacts), so that one cluster buddy is available while database maintenance is being run.

● See IBM Technote: 1170207 for instructions on how to rebuild the CLDBDIR.NSF

Speaker introduction

Securing Your Servers

Server Tuning

Replication

Clustering

Mail Routing

Managing Clients

Wrap-up and Q&A

Agenda

Disaster: Mail Hurricane Rum Punch• Your mail and inbound SMTP servers are constantly maxed out

• Users are complaining of receiving SPAM while valid messages take a long time to deliver

• Your outbound servers got blacklisted and now you can't get outbound mail to transmit

• No mail notifications or probes are configured to tell you if there are bottlenecks or problems

How you got here • The default configuration for a new server is a single mailbox

• Holding undeliverable mail to combat Spam merely overloads the mailbox with dead mail

• No mail notifications or probes are configured to tell you if there are bottlenecks or problems

• Spammers connect to your servers open port 25 and send mail to users without knowing their addresses in advance and typically:

- Server configuration documents accept ‘fuzzy matches’ on usernames

- Group names are accepted for inbound mail

• If your servers are actually sending out spam you will find yourself quickly blacklisted

Recipe for Success: Ingredients

• Enable ‘Internet Sites’ on each server document so SMTP can’t be started by accident

- Setting is on your server document – basics tab-

• In the server configuration document- Restrict connecting hosts if you are using a spam filter

service in the server configuration document

- SMTP Inbound controls: Verify that name exists in a Directory before receiving mail and reject ambiguous names and, if you can, group names

-----

Recipe for Success: Ingredients● Perform anti relay checks for authenticated users

▬ SMTPVerifyAuthenticatedSender=1▬

● Setup your configuration document to only allow “fullname only” addressing

Recipe for Success: Ingredients• In the server configuration document, set number of mailboxes to

be 2, 3 or 4 - Domino will adjust delivery and transfer threads according to

memory availability

• Write an agent to clear dead mail if you want to hold undeliverable mail

• Set up delayed mail notifications to keep on top of routing problems

• Using Notes Named Networks means there are no connection documents and no easy to follow mail topology

• Configure the DDM probes for messaging such as the - “Mail flow statistic check”- “Transfer Queue check”

Chef's Tips

• More mailboxes is not always better. Giving the router additional mailboxes to process can be detrimental to performance

• More transfer threads don’t always do what you think if you haven’t configured multiple concurrent threads to the same destination

• Forcing the server to have more delivery and transfer threads than it thinks it needs, takes resources from other server activity

Disaster: Mail Fidelity Minestrone

• Your users complain of preparing beautifully formatted mail that once delivered outside your organization – it looks completely different!

• Internal messages forwarded between users sometimes lose a lot of formatting

Recipe For Success

• Mail created in Notes or iNotes by your users, with custom formatting and layout shouldn’t be noticeably altered in transit

• Internal mail should retain formatting as it is forwarded and replied to

• You need to explain to your users the behaviour and limitations of mail routing

How you got here

• The default configuration for routing mail externally is to convert to MIME as plain text

• User Mail Files contain mixed message formats- The default setting for internal mail is to be generated and delivered

as Notes Rich Text

- If read via iNotes/Browser it converts to MIME on the fly

- The default setting for inbound mail is that it remains and is stored in MIME format in the user’s mail file

- but if read via Notes it converts to Rich Text on the fly

• The default setting for a Notes user is for an internet addressed message to be converted to MIME by the client as it sends

- Different versions of Notes and Domino will convert differently so a mixed environment will return mixed results

Recipe for Success: Ingredients

• In the server configuration document – Set outbound MIME conversion to HTML or HTML and Plain Text (for older mail systems)

• If your users are on different client versions then set their location documents to send internet mail as rich text (server handles MIME conversion)

• TRY to explain limitations/behavior of mail rendering to users (iNotes v. Notes client, “letterhead”, “mood stamps”)

Chef's Tips• Understanding how mail is stored, formatted and rendered for

reading explains why you will see different effects from different messages and clients.

• User dissatisfaction stems often from unrealistic or misconceived expectations.

• Standardize mail if you have radically different client and server versions and some of them are pre v7.

• Configure clients to send as Rich Text so the server does the conversion.

• Don’t let all servers do MIME conversion, just the newer ones.

• Make sure you allow MIME routing of messages within your domain so mail isn’t converted to Rich Text as it moves between servers.

Speaker introduction

Replication

Server Tuning

Securing Your Servers

Clustering

Mail Routing

Managing Clients

Wrap-up and Q&A

Agenda

58

Disaster: Clients Á La Mode

● A user dropped his laptop in the ocean and now needs to have Lotus Notes restored

●

● You don't have a backup of their local names.nsf, workspace or Notes ID or anything else

59

How you got here● By default local client preferences, the workspace, Notes ID, contacts,

journal or RSS feed information only resides on the local workstation▬ This makes restoring these very difficult, especially if your users are

habitual workspace users, they won't have a reference to their used databases

60

Recipe for Success: Roaming● Roaming allows your users to use any workstation to work from and/or

it can also be used as a backup mechanism for users local data▬ Essentially replicates all local data up to a server▬ Can be used to synch BES data▬

● Users – Tools – People – Roaming = easy to implement, but:▬ Requires “perfectly” working client machines▬ Depends on the users location document pointing to the right server▬ Does prompt the user for input▬ Difficult to clean up roaming data from actual, roaming workstations▬ Best done in conjunction with multi-user client installs

61

Chef's Tips● Anytime you put data on user drives, understand that this presents

another liability/backup situation▬ Server based archives may be easier to manage and change if needed

● If setting up roaming for users in a cluster, be sure that roaming information resides on all cluster mate servers

●

● Be consistent with your roaming folder naming standard▬ Deviations will make finding data and cleanup confusing

● If you cleanup roaming files every x number of days be aware that that could happen in the middle of a client session

●

● ID Vault also allows you to maintain a backup of user ID's▬ If you're in this BP session you're missing the Show and Tell on ID Vault :)

62

Disaster: ECL Mistrust Coladas● Your support desk gets many calls regarding the grey box below and

nobody is sure what to do

● Since nobody EVER was sure how to resolve these client errors, your Help Desk has been telling users to click “trust always”

● This combination has effectively circumvented Domino security and allows anything to be run on client

machines

63

Recipe for Success: Ingredients● Administration ECL – lives in your Domino Directory, gets inherited by

clients the first time they connect to the directory▬ Happens the very first time their client machine gets setup▬ Consists of “what others do”, “using applets” and “using JavaScript”▬

● Security policy settings document – Execution Control List▬ “Refresh” or “replace” can be selected for how to update client ECL's▬ “Once daily”, “when Admin ECL changes” or “never” are possible update

frequencies

64

Recipe for Success: Ingredients● Authorized entities that actually sign code

▬ For example Lotus Notes Template Development/Lotus Notes

● Signing of applications before they get deployed to users▬ Once all of the above is set appropriately, this will be a requirement to deploying new

applications

● Clients existing ECL – file – security – user security – what others do▬

65

Chef's Tips● Create a signing entity for your company and keep it safe

▬ Requiring two passwords on this shared ID is a good idea

▬ This will be used to sign new applications

▬ Don't grant this ID special ACL rights to ensure it can't be used maliciously to run your environment

▬ DO grant this ID rights to run agents on the server▬

● Allowing users to modify their own ECL is risky, but you may need to do so until you have successfully implemented signing ID's

●

● You can only add entities with actual ID files to an ECL▬ Hence groups won't work▬

● You do not need to list specific servers in the admin ECL▬ Unless you have apps signed with server ID's

Speaker introduction

Securing Your Servers

Server Tuning

Replication

Clustering

Mail Routing

Managing Clients

Wrap-up and Q&A

Agenda

67

Q&A ● Franziska Tanner, MartinScott Consulting

▬ Email: francie@martinscott.com▬ Blog: http://www.martinscott.com/blog

● Marie Scott, Virginia Commonwealth University▬ Email: mlscott@vcu.edu▬ Blog: http://www.bleedyellow.com/blogs/crashtestchix

68

Legal Disclaimer● © IBM Corporation 2009. All Rights Reserved.● The information contained in this publication is provided for informational purposes only. While efforts were made to

verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

● References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

●

● IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.

●

● Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

●

● Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.●

● UNIX is a registered trademark of The Open Group in the United States and other countries.● If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete:● Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.● Other company, product, or service names may be trademarks or service marks of others.