Soccnx11 Two wrongs don't make a right - Troubleshooting Connections

Social Connections 11 Chicago, June 1-2 2017

Two wrongs don’t make a right –Troubleshooting Connections

Terri Warren, panagenda

@TerriLLBean

Nico Meisenzahl, panagenda

@nmeisenzahl

PLATINUM SPONSORS

GOLD SPONSORS

SILVER SPONSORS


Nico Meisenzahl

• Consultant at panagenda

• IBM Connections since version 3.0 / 2010

• IBM Notes / Domino since 2008

• Focusing in ICS

• Deployment & consulting

• Optimization and migration

• “panagendian” since 2016

• IBM Champion

@nmeisenzahl

linkedin.com/in/nicomeisenzahl

meisenzahl.org

nico.meisenzahl

+49 170 7355081

[email protected]


Terri Warren

@TerriLLBean

linkedin.com/in/terrillbean

live:TerriLLBean

[email protected]

• Consultant at panagenda (since 2016)

• In my recent past I was a developer on IBM

Connections, IBM Domino

(Directory/Security) and an SRE on Watson

• Frequent presenter at IBM conferences

focusing in ICS

• Deployment & consulting

• Optimization and migration


Agenda

• Howto: Troubleshooting

• Troubleshooting…• Connections itself

• Backend like Spectrum CfC, DB2, TDI, WebSphere, Directory, SSO

• Optional add-ons like Orient Me, Docs, FEB, Cognos, CCM

• Tools


Howto: Troubleshooting


Reproduce the error

• Reproducible and/or periodically?

• A sequence error?

• Client-side or server-side problem?

• Analyze the root cause


Be aware of the big picture

• Client-side problems• Debug in different Browsers (IE, FF, Chrome)

• Do not use a server IE

• Server-side: IBM Connections is based on many components• debug on “high level” first

• get an overview which backend service is causing the error


Configuration changes

• Changes in…

• Connections configuration

• Backend (WebSphere, Database, HTTP, CfC)

• Firewall or network

• OS, hardware or VM

Tip: Even the smallest configuration change can have big consequences!


Analyze log files and browser

• Analyse log files• Atom.io, Notepad++ or less/tail

• Baretail or tail –f

• kubectl logs

• ELK stack

• Tools for client-side problems• Firebug or Developer-Tools

• BurpSuite or Fiddler


Analyze root cause

• Find a hint inside the log

• Network timeout or DNS

• SQL errors

• LDAP errors

• Syntax errors in configuration files „xxx-config.xml“

• Error stack

Tip: In a clustered environment, start and analyze only one

Node (if possible)


Find support

• Knowledge Center https://goo.gl/up6cxG

Troubleshooting Tips https://goo.gl/IaVinx

• IBM Connections Forum http://goo.gl/CVvQCU

• Community Blogs and/or Chats

• Fix Central

• PMR


Troubleshooting


WebSphere Application Server logs

• SystemOut.log

• SystemErr.log

• trace.log (if tracing is enabled)

• Log path:• <wasroot>/profiles/<profilename>/logs/<servername>/


Analyze WAS log files

• Time stap: 24h time stamp with milli-seconds

• Thread id: eight character hexadecimal value

• Short name: typically java class name

• Event type: one character only (E, W, I,…)

• Message identifier: String based on component

• Message: Some information


WAS Event types

• F - Fatal message

• E - Error message

• W - Warning message

• A - Audit message

• I - Informational message

• C - Configuration message

• D - Detail message

• O - Messages that are written directly to System.out by an application

• R - Messages that are written directly to System.err by an application

• Z - Place holder to indicate type was not recognized


WAS Message identifier

• Prefix by Application or Server (CLFRW)

• Specific application code (0042)

• Event Type (I)


Read trace stack

• First line displays key information

• “Caused by” displays root cause


Environment information

• First log lines on server startup

• WebSphere version

• OS version, Process id

• Installation path

• Java version


Enable tracing

• Enable tracing using ISC

• Runtime or configuration

only

• Define tracing based on

• App prefix / error stack

• Must gather (PMR)


Search issues

• http(s)://<fqdn>/search/serverStatus

• Display index, seedlists, log information

• Data is displayed for one node only• Using node fqdn to access different nodes

Tip: This will create persistent files on your hard disk!


Debug Search

• Search queries (runtime)

• com.ibm.connections.search.index.searching.*=all

• Crawling & seedlists

• com.ibm.connections.search.index.indexing.*=all:

com.ibm.connections.search.seedlist.*=all:

com.ibm.connections.httpClient.*=all


Recreate Search Index

• SearchService.startBackgroundIndex()

• Crawls seedlists

• Extracts the file content

• Create index

Tip: Use „all_configured“ to index all apps


CLFRW0394E: Search indexing of services

...

• Search index not ready

• interruption at index creation• CLFRW0283E: Search has encountered a problem

while crawling

• CLFRW0027E: Error Indexing component <app> for search

• INDEX.READY file not present

• Recreate and enable tracing


Database connections

• Check datasouces• ISC – Resources – JDBC – Data sources

• Check logs for more information

• DB2 server log• <instanceroot>/sqllib/db2dump/

• db2diag.log

• db2diag.xxx.log (log rotation, you should enable this!)

Tip: Oracle users have password expiration enabled by default!


HTTP Server (IHS & Plugins) logs

• IBM HTTP Server

• <installroot>/logs/

• error_log

• access_log

• based on configuration

• WebSphere AppServer Plugins

• <installroot>/logs/<webserver>/http_plugin.log


HTTP 404 not found

• Outdated Plugin configuration

• Restart IHS

• WAS Plugin configuration issue

• http_plugin.log


HTTP 404 not found

• AppServer or App down

• Network issue

• http_plugin.log


HTTP 500 Internal Server Error

• Unexpected error

• http_plugin.log

• Configuration issue

• WAS Root certificate not trusted or missing

• SSL certificate expired


Debug wsadmin

• Enable trace within wsadmin session

• AdminControl.trace(‘com.ibm.*=all’)• <wasroot>/profiles/<profilename>/logs/wsadmin.traceout


Directory/Troubleshooting/ Tuning

• Waltz and Sonata overview

• Enabling Log parameters…

• Cache Tuning

• SSO Config Glitches

• CCM Troubleshooting


Connections Troubleshooting

• Waltz: "common directory services"

• Directory Service eXtension (DSX) REST API.

• Sonata: common HttpClient

services through a RESTful service

• configurable authenticators that support

• SSO security tokens and cookies for

secure server-to-server communication.


Troubleshooting:

DSOutOfServiceExceptions

• Symptoms:• Access Connections not possible

• DSX not working


Troubleshooting: Enable Logging

• Track all basic Waltz configuration settings and transactions.• com.ibm.connections.directory.services.*=all

• Track all LDAP transactions in between Waltz & LDAP server(s).

• WaltzLDAPUsage=all

• Track all LDAP entry to be cached and hit from cache upon DN of LDAP entries.

• WaltzDNEntryCache=all

• Track all LDAP entry to be cached and hit from cache upon ID of LDAP entries.

• WaltzExactIDMatchCache=all• Track all group membership (a list of groups) for a given user upon ID.

• WaltzGroupMembershipCache=all

• Track all members (a list of users) for a given group upon ID.

• WaltzMemberExpansionCache=all

TIP: Save runtime changes to make changes persistent



• WAS VMM/WIM:

• com.ibm.websphere.wim.*=all:com.ibm.ws.wim.*=all

• WAS SPNEGO:

• com.ibm.ws.security.spnego.*=all

• Apache commons HttpClient:

• org.apache.commons.httpclient.*=all

• WAS security:

• com.imb.ws.security.*=all:com.ibm.ws.security.policy.*=off;



• Basic Sonata configuration settings and transactions:• com.ibm.connections.httpClient.*=all"

• Track the amount of HTTP(S) transactions.• SonataHttpUsage=all

• Track headers received for all HTTP(S) transactions.• SonataHttpHeader=all

• Track all bodies received for all HTTP(S) transactions.• SonataHttpBody=all


EJPVJ9284E: Unable to get the groups

from the directory for the user…

• User was not able to access Connections

anymore

• WAS LDAP bind user had no read access

to one of the groups the user was member


SSO Configuration Glitches:

• Warning indicating SSO config is missing or

incomplete enabling application security:

• Ensure you filled

in your domain

Tip: Connections CRs sometimes

resets SSO domain


SSO Configuration Glitches:

• LtpaToken Cookies: • More secure- recommended to use version 2 LTPA cookies

In other words- interoperability mode is left as disabled by default.

Only enable interoperability mode ifsome app servers are still relying on LTPA version 1 and need to interoperate.

Connections Security supports custom LTPA cookie names since 4.5 CR1

Sets com.ibm.websphere.security.customSSOCookieName custom properties (verify 3rd party products Portal/Domino can handle it)


Waltz Cache: Changing Group

Membership/permissions remains in Waltz Cache

• Issue: Administrator changes access or Group membership for a user. That user does not get immediate access.

• Cache: Connection's Directory Services keeps a “list” of directory objects it has already searched for performance reasons

● Improves performance of connections services

● Reduces load on remote directory servers

● Cache is based on a timing mechanism

● Cache is "flushed and renewed" on a 12 hour time schedule


Tuning the Cache: Continued

• How does this affect Group Membership?● Group membership exists in the cache for 12 hours

● Operations such as renaming, deleting, or updating groups remain in the cache for that time

• Configuration parameter for cache timing mechanism

● Enables Connections administrators to flush the cache in configurable time increments

● Set cache parameter via JVM arguments to flush them (we'll get to that!)

● Default remains 12 hours


Cache Configuration: Setting the cache

flush parameters• Setting the cache parameters via JVM configured in

WebSphere● Google Setting generic JVM arguments in WebSphere Application

Server

• Time Values must be the same for all cache settings

• Adjust with care! Flushing the cache has implications!


IBM Connections Content Manager

(CCM): Troubleshooting• During library creation, retrieving or modifying content:

• Calls are made between Communities, Profiles, and FileNet Issues with Filenet / Connections Profile or Community configuration can

adversely affect access to Libraries and CCM

Logging from Connections components (previous slides) and FileNet essential

• SystemOut.log & trace.log – these are your friend

• FileNet core logging: • <wasprofile>/<servername>/p8_server_error.log

• <wasprofile>/<servername>/p8_server_trace.log

• <wasprofile>/<servername>/pesvr_system.log

• <wasprofile>/<servername>/pesvr_trace.log


CCM Troubleshooting: Filenet URLS

• Status about FileNet Collaboration Services:

• check the status page at http://<fqdn>/dm

Navigator version and configuration

• Status about FileNet Content Engine

• check the status page at http://<fqdn>/FileNet/Engine

Server status, version, sonata/waltz version

• Status about FileNet Content Engine Domain and Object Store

• check the status page at http://<fqdn>/P8CE/Health

Health checks for authentication, stores and database


Debug CCM Widget

• Widget issues

• com.ibm.quickr.communitylibrary.*=all:

com.ibm.lconn.widgets.service.*=all:

com.ibm.lconn.widgets.actions.*=all

• Authentication issues (Remember those cool tips a few

slides back? )

• com.ibm.connections.directory.services.*=all:

com.ibm.connections.httpClient.*=all


Debug FileNet using ACCE

• ACCE tool: check if FileNet Content Engine can

authenticate using Connections and inspect other data

for errors:

http(s)://<fqdn>/acce


CCM: Enable logging for Connections and FileNet

using JVM Properties

• Download and copy the sample waltz.sonata.log4j. as log4j.xml:• Linux: /<absolute_path_for_log4j_file>/log4j.xml

• Windows:\<absolute_path_for_log4j_file>\log4j.xml

• Add generic JVM properties

• -Dlog4j.configuration=file:<path>/log4j.xml -DskipTLC=true

• Tells the Content Engine (CE) server to skip it’s tracing configuration and writes to waltz.sonata.trace.log Windows: c:\Program Files\IBM\WebSphere\AppServer\profiles\AppSrv01Linux: /opt/IBM/WebSphere/AppServer/profiles/AppSrv01


CCM: Enable logging for Connections and FileNet

using JVM Properties

• Copy & customize sample log4j.xml

• <ContentEngineRoot>/config/samples

• 20 subsystems (db, engine, security, search,…)


TDI logs

• <tdisol>/log/ibmdi.log• TDI log file

• <tdisol>/employee.*• Files include all changed users (adds, update, delete, error, skip)

• <tdisol>/syncupdates/*• Temporary files within the sync

• Including database dump and ldiff

• sync_updates_clean_temp_files=false (default: true)• profiles_tdi.properties

Tip: Check the lock file


Analyze TDI logs

• Error code prefix• CLFRN: Profile & User synchronization

• CTGDIS: TDI itself

• Error code suffix

• I, E, W, …


Debug TDI

• Profile & User synchronization

(<tdisol>/etc/profiles_tdi.properties)

• source_ldap_debug=true

• debug_update_profile=true

• debug_collect=true

• TDI issues (<tdisol>/etc/log4j.properties)

• log4j.rootCategory=DEBUG, Default


Cognos BI logs

• Cognos BI• SystemOut.log & trace.log

• <installroot>/logs/cogserver.log

• Cognos Transformer• <installroot>/logs/cogserver.log

• PowerCube build• <installroot>/metricsmodel/trxschelog.log

• <userhome>/Transformer/Logs/*.log (win only)


BMT-MD-6003 No connection to the

data source …

• PowerCubes not created yet

• Check cronjobs or scheduled jobs

• trxschelog.log


Debug Cognos BI & Metrics

• Communication between Cognos BI and Connections Metrics

• SonataHttpUsage=all:SonataHttpHeader=all:SonataHttpBody=all: com.ibm.connections.httpClient.*=all: com.ibm.connections.metrics.*=all

• Connections Metrics Servlet

• com.ibm.connections.metrics.cognos.servlet.*=all


Docs/Viewer logs & urls

• SystemOut.log & trace.log

• http(s):<fqdn>/vsanity/check

• http(s):<fqdn>/sanity/check?app=all&querytype=report

• http(s):<fqdn>/*/version


Debug LTPA between Domino & WAS

• Debug on Domino side (notes.ini)

• Debug_SSO_Trace_Level=2

• Webauth_verbose_trace=1

• WebSess_verbose_trace=1

• Debug_outfile=<logfilepath>

• Debug on WebSphere

• com.ibm.ws.security.ltpa.*=all


Debug Kerberos

• Configuration• com.ibm.ws.security.spnego.*=all:

com.ibm.ws.security.*=all: com.ibm.issw.spnegoTAI.*=all: com.ibm.security.krb5.*=all

• Runtime• com.ibm.connections.httpClient.*=all:

com.ibm.connections.directory.services.*=all:com.ibm.websphere.wim.*=all:com.ibm.ws.wim.*=all

• Fiddler & BurpSuite


IBM Spectrum CfC stats

• https://fqdn:8443


Docker & Kubernetes status

• kubectl get pods (--all-namespaces)

• kubectl get services

• kubectl get deployment

• kubectl describe pods xxx

• docker ps

• docker info


Docker & Kubernetes status

• Persistent Volume status

• kubectl get pv,pvc


Orient Me logs

• Display application logs

• kubectl logs xxx• -f

• --tail=20

• --since=1h

• Special tasks for MongoDB and Solr


MongoDB Troubleshooting

• Check persistent volumes

• Access logs

• kubectl logs xxx [mongo|mongo-sidecar]

• Get replica status

• kubectl exec -it mongo-0 -- mongo mongo-0.mongo:27017 --eval

'rs.status()’

• Check “health” status for all 3

members


Solr logs

• kubectl exec -it xxx -- cat /var/solr/logs/solr.log

• kubectl logs xxx

• only display deploment logs


Pods startup issues

• kubectl describe pod xxx


CfC logs

• docker logs -f kubelet

• docker logs -f calico


Docker tracing

• service docker stop

• docker daemon –debug

• journalctl -u docker.service


What to get more insights on Orient Me?


Tools


Analyze logs

• Analyze logs live• Baretail

• tail –f

• View logs • Atom.io, Notepad++

• less, tail

• ELK Stack• Elasticsearch, Logstash, Kibana

• Small Docker deployment

• Kubectl (local client)


Analyze logs

• Kubectl (local client)


Analyze Client-side

• Browser

• Firebug / Developer Tools

• Intercepting proxies

• Fiddler

• Burp Suite

• VMs with different IE versions

• Without GPO

• https://www.modern.ie/en-us/virtualization-tools


Database Clients

• db2 command

• Dbeaver or IBM Datastudio

• Robomongo (MongoDB)

Tip: Database write access is not supported!


LDAP & Search

• LDAP

• ldapsearch command

• Softerra LDAP Browser

• Apache Directory Studio

• Search Index

• Luke (Lucence Index Toolbox)


Analyze Network

• Wireshark

• tcpdump


Q&A

PLATINUM SPONSORS

GOLD SPONSORS

SILVER SPONSORS

Technology

Soccnx11 Two wrongs don't make a right - Troubleshooting Connections