Adm07:The Health Check

Extravaganza for Social and Collaboration Environments

Kim Greene, Kim Greene Consulting, Inc.

Luis Guirigay, IBM

1#engageug

2#engageug

Kim Greene - Introduction• Owner of Kim Greene Consulting, Inc.

• 15+ years experience with Domino and

Sametime and 20+ years of experience with

IBM i (AS/400, iSeries)

• Services include System & Application

performance optimization, Administration,

upgrades, health, performance, security etc.

checks, migrations, custom development,

enterprise integration

• IBM Champion

• Blog: www.dominodiva.com

@iSeriesDomino http://www.kimgreene.com

Luis Guirigay - Introduction

3#engageug

• WW Executive IT Specialist

• Global Technical Ambassador at IBM

• Published Author. IBM Redbooks and

developerWorks (Domino, DB2, iSeries,

Connections, Sametime)

• IBM Certification Exams for ICS Products (writer

and reviewer)

• WW Program Manager for Project Hawthorn

(mail support for MS Outlook)

• SME – Social, Collaboration, Cloud, Verse and

Messaging

Follow me @lguiriga or http://about.me/lguiriga

Agenda

4#engageug

• Proper maintenance

• Configuration & best practices

• Keeping current a.k.a patching

• Monitoring

• Security

• SmartCloud Notes Hybrid

• IBM Mail Support for MS Outlook

PROPERMAINTENANCE

5#engageug

Many Moving Parts – What to do?

6#engageug

• Modern Collaborative Systems have many moving

parts but which ones are most in need of

maintenance and how?

• Let’s look at some of the Systems and all their

Moving Parts

The Bits and Pieces of IBM Domino

7#engageug

These are the individual moving parts that make up your

IBM Domino environment:

Main Components:

● Servers (the OS)

● Server function

(application, mail, traveler,

etc.)

● Domino NSFs

Possible Additions:

● Transaction logs

● .NLOs

● DB2

● Third party products /

applications

.NSF Maintenance

8#engageug

Updall

• Updates view indexes

• Runs at 2AM by default

Fixup

• Check integrity of Domino databases

• Resolve corruption problems

• Especially important if not using transaction

logging

DBCapture Tool

9#engageug

Automatic identification and collection (i.e. taking them off-line)

of corrupt databases without bringing down Domino server• Files renamed to .cor and moved to IBM_Technical_Support folder

• Can still run fixup / compact / updall on them!

Enable using server notes.ini: • DATABASE_CAPTURE_ENABLED=1

DBCapture Tool

10#engageug

Tips:Can invoke manually; ignores Status but respects Capture and Size limits

• load dbcapture dbnames.nsf

DATABASE_CAPTURE_SIZE_LIMIT (in mbs) sets size of all collected

databases• Default: 100 / set to 0 for no limit

DATABASE_CAPTURE_LIMIT sets maximum # corrupt DBs to be collected• Default: 10 / set to 0 for no limit

Gotcha• DATABASE_CAPTURE_ENABLED value resets every time capture is done, and is enabled again

when server is restarted! (i.e. does not run continuously)

.NSF Maintenance

11#engageug

• Compact• Equivalent to a “Defragment” for a Domino database

• Rearrange database or reduce file size

• Run with multiple threads via notes.ini

• debug_enable_compact_8_5=1

OR

• Use DBMT

• Recent customer exampleCompacted databases after upgrade, recovered 418 GB of disk space, a 42%

reduction!!

Compact Tips

12#engageug

compact_filter=dbname.nsf• Prevents compact running on specific databases

• Ex: compact_filter=log.nsf, names.nsf, admin4.nsf

> load compact -c mail/ladmin.nsf

Database 'mail/ladmin.nsf' is not present in the ini

parameter 'COMPACT_FILTER'. Proceeding with compact.

• Compact –ODS• Copy style compact if current ODS is less than desired level

• 95% Space Utilization is a good thing

Compact Replication

13#engageug

• Use –REPLICA switch on Compact command• Creates replica of database under the covers while source database remains

accessible

• Use to remedy “Insufficient memory” or “Unable to extend an

ID table – insufficient memory” errors caused by frequent

additions and deletions in a database

• Internally reorganizes IDs in new replica

• Avoids ID table fragmentation leading to above errors

• Preventative maintenance to avoid fragmentation causing database to become

inaccessible

• Maintains Views and Unread Marks between old and new replica

Derby Database Maintenance

14#engageug

• Over time Traveler performance can deteriorate, defrag to

restore performance

• Steps to start a Defrag manually:

• Tell traveler shutdown

• Tell http quit

• Load traveler -degrag

• Notes.ini variables

• NTS_DEFRAG_INTERVAL_DAYS=<# of days>

• NTS_LAST_DEFRAG=<timestamp of last defrag>

Traveler Database Maintenance

15#engageug

Defragging changes in 9.0.1.8 and later versions of Traveler, use DBMAINT now• Tell traveler dbmaint set interval 7 **

• 11/19/2015 09:37:02 Traveler: DB maintenance will be performed every 7 days.

• Tell traveler dbmaint set time 23:00• 11/19/2015 09:39:58 Traveler: Time of day for DB maintenance has been set to

23:00

• Tell traveler dbmaint set day Sunday• 11/19/2015 09:51:40 Traveler: Day is now configured to Sunday.

• Tell traveler dbmaint set auto on **• 11/19/2015 10:12:27 Traveler: Automatic maintenance of your database has been

set.

• 11/19/2015 10:12:27 Traveler: The next maintenance is scheduled for 2015-11-22

23:00.

• 11/19/2015 10:12:27 Traveler: Maintenance will be performed every 7 days at

23:00.

• ** Only options available for Derby database

The Bits and Pieces of IBM Connections

16#engageug

These are the individual moving parts that make up your IBM

Connections environment:

Main Components:

● Servers (the OS)

● WebSphere

● DB2

● LDAP

● IHS

● TDI

Possible Additions:

● Cognos

● IBM Docs / IBM FileViewer

● IBM Forms/Surveys

● Third Party Products

● ICMail

● Shared File Space (NAS/NFS,

etc.)

The Bits and Pieces of IBM Sametime

17#engageug

These are the individual moving parts that make up your IBM

Sametime environment:

Main Components:

● Servers (the OS)

● WebSphere

● DB2

● LDAP

● Domino (Community

Server only)

Possible Additions:

● Proxy Servers

● Integrations with Voice/Video

devices

● Integrate Sametime with other

systems (awareness, meetings, etc.)

● Third Party Products – IM Queue

Managers, etc.

VPUserInfo.nsf - A Contact List Tune-Up

18#engageug

• Vpuserinfo.nsf can grow very large and make Sametime

very slow to login and respond to searches.

• Use a custom agent to look for users no longer registered in the Domino

Directory and remove all contact lists for those users.

• If users are seeing partial empty lists:• load fixup vpuserinfo.nsf

• load updall -r vpuserinfo.nsf

• load compact vpuserinfo.nsf

DB2 - Three “Rs” rule

19#engageug

• Reorganisation• Recommended after large amounts of data get added

• Runstats• Run often to make sure queries are being executed optimally

• Rebind• Recommended after applying a fix pack or similar

CONFIGURATION & BEST PRACTICES

20#engageug

The Tyranny of the “Default”

21#engageug

• Everyone gets an “average” server if they do

nothing at all

• It will run, but will it run well?

• Is this acceptable to you?

Connection Documents

22#engageug

• Key for properly controlling replication

• What to replicateReplication type and files / directories to replicate, and avoid

• Tip: Are you replicating names.nsf, admin4.nsf, events4.nsf and dir

assist db throughout the domain??

• Replication time limit• Tip: Set to less than the repeat interval

Connection Document Settings

23#engageug

• Critical to watch connection document settings

• Customer example• 09/26/2014 11:52:33 AM {User ABC/ACME} DBStore::GetDB:

Unable to open CN=DomMail/O=ACME!!mail\uabc.nsf (Connection

denied. The server you connected to has a different name from the

…

• Connection document was culprit

IP of Source server, not Target!

Notes.ini Files

24#engageug

• Is there lurking debug still enabled?• Did you really check??

• Consumes valuable resources

• Make sure your notes.ini doesn’t look like this• Debug_threadid=1

• Log_AgentManager=1

• Debug_sem_timeout=10000

• Log_update=2

• NSF_DocCache_Thread=1

• debug_nif=0

• Debug_nif_update=1

• FT_LIMIT_HIGHLIGHT_FILTER=1

• LDAPDEBUG=1

• SMTPDebug=3

Notes.ini Files

25#engageug

• Don’t forget about the configuration document!!

Notes.ini Files

26#engageug

• Recommended to be enabled at all times:

• CONSOLE_LOG_ENABLED=1• Captures server console data and logs to console.log file

• CONSOLE_LOG_MAX_KBYTES=204800• Restricts the console Log size to 200MB and then overwrites

oldest entries

• DEBUG_THREADID=1 • Stamps server threads and logs to console.log file

• DEBUG_CAPTURE_TIMEOUT=1• Captures semaphore time stamp and logs to semdebug.txt

• DEBUG_SHOW_TIMEOUT=1• Captures semaphore information and logs to semdebug.txt

Transactional Logging

27#engageug

• Has been around for years!

• Sequential writing into a log file

1 2 3 4 5 6 7 8 9 …

Remove unread mark

Remove unread mark

vs.

• Allows Incremental Backup/Restores

• TXN Logs stored on a separate disk controller for best performance

(depending on your platform)

Traveler HTTP Threads and Sizing

28#engageug

• Tell Traveler stat show push.devices.total

Push.Devices.Total = 225

• This indicates that 225 devices are registered for

synchronization with the Notes Traveler server and

that at least 270 HTTP threads are needed (1.2 x 225 =

270).

Tip: The number of active HTTP threads needed for Traveler

is calculated this way: 1.2 x Number of registered devices =

Number of needed active HTTP threads

Sametime MUX

29#engageug

• Geographic

• Go from 20,000 users to 100,000 per Community

IBM Connections

30#engageug

• DB2: 64 Bits, 8GB -128GB

• Dedicated Storage or high

performance disk

• Use a Caching Proxy Server

https://ibm.biz/BdHCUh

• DB2 Pool Size

• Content Compression

• More Tuning Tips

https://ibm.biz/BdHC5j

KEEPING CURRENT A.K.A PATCHING

31#engageug

On Disk Structure

32#engageug

• Don’t forget to upgrade databases to latest ODS level

when upgrading servers• What is the ODS about?

• Newest internal structure enables database to benefit from

newest features

• Examples of benefits • R5.0 (ODS41) = participate in transaction logging

• R6.0 (ODS43) = LZ1 compression and shared templates

• R8.0 (ODS48) = design and document compression

• R8.5 (ODS51) = DAOS

• R9.0.1 (ODS52) = Performance improvements, better handling of huge

(2GB+) attachments

How to Upgrade On Disk Structure

33#engageug

• For server• Copy style compact (compact –c)

• Remember compact -ODS

• For client• Use policies to update local ODS levels

• Push to clients via dynamic policies / or organizational policies• Desktop Settings policy document: Mail tab > “Enable upgrade for all local

NSFs to latest ODS version”

• Gotcha: requires the 8.5.2 Domino Directory on server

• CREATE_R(85/R9)_DATABASES=1

• Even better: NSF_UpdateODS=1 (Will keep updating ODS levels

as new versions are released) • Tip: Although it’s said to be both server & client side, it only works on the

client side!

Preventing ServerTasksAt Updates

34#engageug

• Tired of losing your ServerTasksAt customizations when

upgrading?

• SetupLeaveServerTasks to the rescue• Add SetupLeaveServerTasks=1 to server’s notes.ini

• Disables automatic updating of ServerTasksAt#= lines during a Domino

Server upgrade

MONITORING

35#engageug

Key Items To Keep In Mind When Monitoring

36#engageug

All systems require you to cover the basics for all servers involved:

CPU, Memory, Disk, Network

When monitoring:

• Make it actionable

• Know your baseline

• Know what your results mean

• Investigate!

Monitoring for Domino

37#engageug

• Pay attention to console messages, don’t ignore them!

• admin4.nsf has not replicated (PUSH) with ANY server since

MM/DD/YYYY HH:MM:SS (1681 hours ago)

• Error validating execution rights for agent 'Notify' in database

‘subdir/dbname.nsf'. Agent signer ‘XXX01/YYY', effective user

‘XXX01/YYY'. Agent signer.

• RnRMgr: The design of Rooms.nsf is not one supportable by

RnRMgr. Autoprocessing is being disabled for this DB.

• Directory Cataloger finished processing DirectoryCatalog.nsf: File

does not exist

• Agent Manager: Full text operations on database ‘mail/myfile.nsf’

which is not full text indexed. This is extremely inefficient.

Monitoring for Domino

38#engageug

• Health Monitor

• Easy to use and provides 24/7 monitoring

• Enabled via Administration Preferences

Monitoring for Domino

39#engageug

• Health Monitor

• Watch servers on single screen

• Monitor servers and/or tasks needing attention

Monitoring for Domino

40#engageug

• tell traveler status

Example Yellow status

• Example Green status

IBM Connections

41#engageug

• CPU Utilization on WAS• If > 70% for 5 minutes or longer = too high

• CPU Utilization on DB2• If >50% for 5 minutes or longer = too high

• Look for these words in SystemOut log

• “Hung”

• “Starvation”

SECURITY

42#engageug

Security and Collaboration Systems

43#engageug

IBM Connections, Sametime and Domino are made up of individual

components that all have separate security concerns and (potential)

vulnerabilities.

No system will be 100% secure. If Your Domino/Connections/Sametime

environment were your home, what you would look for:

1. Every door of your house has a lock and a deadbolt and every

window can be closed and locked.

2. You would not leave a key under the front mat or in the flower pot

next to the door.

3. No Notes sticking on the front door detailing which flowerpot to look

under for the key.

4. You would have a security light or two and maybe a warning sign of

the dangerous attack Chihuahua dog that lives in your house . . .

Security: Common Sense Questions to Ponder

44#engageug

1. Do you really want to use the same system/generic account for each

function?

2. Do you really need the “One Admin Account to Rule Them All”?

3. Do you have so many admins that creating individual admin accounts for

them is a great administrative overhead?

4. When assigning rights, are you thinking of “person” or of “job function”?

5. Do you have more than one “person” or “admin type” for each function so

you have continuity?

6. Is your brilliant administration scheme actually documented someplace?

7. If you use hierarchical directories (LDAP …, it’s hierarchical) are you taking

advantage of it?

Domino – Protected Groups

45#engageug

• Prevents accidental deletion of designated “critical” groups

• Configured in Directory Profile of the Domino Directory

• Tip: You must edit and save once to become operational

• Requires Domino directory to have 9 design

• Defaults to LocalDomainAdmins, LocalDomainServers, and

OtherDomainServers

Domino – Protected Groups

46#engageug

• Open Domino Directory→Actions→Edit Directory Profile

Domino – Protected Groups

47#engageug

• Prevent deletion of these groups

Internet Access to Domino

48#engageug

• Oldie but goodie.....PASSTHRU SERVERS!!!

• Separate Domino Domain

• Configuration Only Names.nsf

Lock Down Ports

49#engageug

• Lock down ports not using

• Number one step for outside attacks

• Nmap is great tool for testing open ports

Lock Down Ports

50#engageug

• Ports commonly seen openPortPortPortPort FunctionFunctionFunctionFunction

25252525 SMTP

80808080 HTTP

85858585

110110110110 POP3

113113113113 Authentication service

143143143143 IMAP

179179179179 Border gateway protocol

389389389389 LDAP

443443443443 HTTP SSL

465465465465 SMTP SSL

541541541541 uucp-rlogin

Fortimanager and Fortigate server

587587587587 Alternate outgoing SMTP

993993993993 IMAP SSL

995995995995 POP3 SSL

1352135213521352 Notes remote procedure call

2050205020502050 Java server console

1503150315031503 Sametime meeting server listen

1533153315331533 Sametime community server listen

8081808180818081 Alternate HTTP port

60000600006000060000 DIIOP

63148631486314863148 Remote debug manager

Lock Down Ports

51#engageug

• Lock down at firewall level

• To prevent getting to server

• Lock down at server level

• In case firewall is not secured properly

• Is LDAP, POP3, IMAP, DIIOP, etc. in use?

• Enabled by default

ID Vault

52#engageug

• It’s a vault with a secured/encrypted copy of all user ids

• You can have multiple vaults

• Important: Do not use standard replication for ID Vault

replicas

• Some of the benefits are:

• Lost or forgotten user passwords can be recovered or

reset easily

• User renames and key rollovers are automated

• User IDs are synchronized across machines

• No need to carry ID files for new installs

• Corrupted IDs are replaced automatically

IBM MAIL SUPPORT FOR MS OUTLOOK (HAWTHORN)

53#engageug

Functionality Today

54#engageug

• Primary Domino communication via HTTP

• Exchange ActiveSync synchronizes all data

• Mail, calendar, contacts, folders

• REST services:

• Out of office

• Encryption

• Room finder

• Quota management

• Delegate management

• Address book search via LDAP

• Native Outlook capability

• Any LDAP will work (not just Domino)

Architecture Guidelines

55#engageug

• Outlook users must have a replica on the IMSMO servers

• Second IMSMO Server required for HA via Outlook

• Load Balancer is also required. Outlook is dumb!

• You can build a cluster with IMSMO and non-IMSMO servers

• You can use the same DB2 server to host multiple DB2

instances

• Think one DB2 server for multiple IMSMO clusters

• You must use a proper SSL certificate

SMARTCLOUD NOTES HYBRID

56#engageug

Architecture

57#engageug

Questions

58#engageug

Thank You!!