GWAVACon 2013: Novell Open Enterprise Server Best Practices

Novell® Open Enterprise Server Best Practices

Martin Weiss Peter ReckSenior Architect Infrastructure Lead Architect Infrastructure Solutions

[email protected] [email protected]

Madhan P.OES Product Manager

[email protected]

mailto:[email protected]



© Novell, Inc. All rights reserved.2

Agenda

Things to Think About

Installation

Configuration

Administration

Troubleshooting

Questions and Answers

Novell® Open Enterprise ServerThings to Think About


Naming StandardsNovell® Open Enterprise Server (OES)

• Enhance your naming conventions forcase-sensitivity

‒ *ix is case sensitive

• Implement uniqueness‒ Names of LUM enabled users and groups must

be unique across the tree

‒ iManager unique naming plugin is your friend

‒ make sure that uniqueID = CN and both are single valued

• OES server names in lower case


LAN ConnectivityNovell® Open Enterprise Server

• Use bonding driver for fault tolerance

• Link monitoring‒ MIIMON is likely to always return “Up” in blade centers

‒ Use ARP ping to the default gateway instead

‒ Increase the polling interval to 1000 ms

• Bond mode‒ Active-backup or 1 = fault-tolerance

‒ Pre-define the default primary (primary=eth0)

‒ Various modes for load sharing (optional)‒ some modes require switch configuration

‒ /usr/src/linux-<kernel-version>/Documentation..../networking/bonding.txt


SAN / Storage ConnectivityNovell® Open Enterprise Server

• Use DM-MPIO wherever possible‒ Adjust timeout values to your needs (cluster nodes)

‒ dev_loss_tmo=<n>, fast_io_fail_tmo=<n>(/etc/multipath.conf)

‒ Use user_friendly_names stored in /var/lib/multipath/bindings

‒ Move to “root” partition if /var is on a separate partition change in /etc/multipath.conf → bindings_file /etc/multipath/bindings

• NLVM.CONF‒ /etc/opt/novell/nss

‒ include/exclude devices, debug settings


File System Design (1)Novell® Open Enterprise Server

• Use separate devices for system and data

• Use GPT partitioned devices for redundancy

• Never use the same device for POSIX file systemsand NSS file systems


File System Design (2)Novell® Open Enterprise Server

• System device‒ primary partition for /boot; ext2/ext3; min. 200 MB

‒ primary partition for LVM (VG system); remaining capacity‒ /swap as much as there is memory; max. 4GB

‒ / ext3, 10GB

‒ /var ext3, 3-5*memory + 10 GB

‒ /tmp ext3, 5 GB

‒ SLES 11 has only 50% of the inodes in ext3 that were in SLES 10 (TID 7009075)

• Data devices‒ POSIX: use LVM for flexibility

‒ NSS: 1 device – 1 partition – 1 pool (segment)


Time Synchronization (1)Novell® Open Enterprise Server

• Always ensure all your servers are synchronized tothe same time source

• Convert your existing timesync environment to NTP

• Use external clocks to ensure accurate time

• Implement a hierarchical, fault tolerant time providerstructure

‒ servers on the top layer will use external time sourcesand will be NTP peers to each other (like Reference servers)

‒ servers on the second layer will use the servers on thetop layer as time source and will be NTP peersto each other (like Primary servers)

‒ all other servers will consume time from at least two serverson the second layer (like Secondary servers)


Time Synchronization (2)Novell® Open Enterprise Server

• Use burst and iburst to speed up time synchronization

‒ http://www.novell.com/coolsolutions/feature/15345.html

• Set HWCLOCK="--localtime" in /etc/sysconfig/clock

• Set NTPD_FORCE_SYNC_ON_STARTUP="yes" and NTPD_FORCE_SYNC_HWCLOCK_ON_STARTUP="yes" in /etc/sysconfig/ntp


Name ResolutionNovell® Open Enterprise Server

• DNS‒ same as on NetWare®

‒ ensure servers can be resolved before you install (also reverse)

• Hosts‒ same as on NetWare®

• SLP‒ move to openSLP

‒ persistent service registration is available since May 2010

‒ /etc/slp.conf:

‒ net.slp.dasyncreg = true/false

‒ net.slp.isDABackup = true/false → /etc/slp.reg.d/slpd/DABackup

‒ net.slp.DABackupInterval = time_in_seconds

‒ net.slp.DABackupLocalReg = true(cannot be configured through YaST)

Novell® Open Enterprise ServerInstallation


Installation (1)Novell® Open Enterprise Server

• Prepare the environment for the first OES server in your eDirectory tree

‒ OES services design (i. e. LUM)

‒ versions, patches, schema

‒ time synchronization, eDirectory™ synchronization

‒ do a full eDirectory health check (TID 10060600)

• Do a SDI health check (TID 3455150)‒ use TKinfo to analyze SDIDiag output files

(http://www.novell.com/coolsolutions/feature/16544.html)

• Do a PKI health check (TID 7000654)‒ verify CA and SSL certificate lifetime

‒ renew certificates depending on lifetime

http://www.novell.com/coolsolutions/feature/16544.html


Installation (2)Novell® Open Enterprise Server

• Use AutoYaST to install your servers

• Deploy SLES and OES updates during installation(YUM repositories)

• 1. Install, 2. Patch (during install), 3. Configure

• Use ZCM to configure/update your servers

• Only install what is really required (pattern based)‒ do never install an individual package

(i. e. novell-imanager.rpm) when there is a corresponding pattern (Novell iManager)

• Check out the Novell Consulting Best Practice Guide that is part of the OES documentation


Installation UserNovell® Open Enterprise Server

• Installation user‒ will be the first LUM enabled user of your tree

‒ required for OES configuration in YaST

• admingroup‒ OES11: automatically created in the context of the installation user

‒ OES11 SP1: can be selected during installation

‒ will be the first LUM enabled group in your tree

‒ installation user will be a member of this group

‒ all workstation objects for servers installed by this userwill be members of this group

‒ will control LUM enabled services

• Recommendation

‒ consider using a dedicated installation user

‒ place high in the tree, i.e. in cn=OESInstall.ou=Services.o=<Org>

Novell® Open Enterprise ServerConfiguration


OES LDAP ServersNovell® Open Enterprise Server

• “LDAP Configuration for Open Enterprise Services” in YaST serves as a template for the LDAP configuration of the OES services

‒ having multiple OES LDAP Servers configured does NOT mean fault tolerance or load balancing!

‒ only servers configured here can be selected for an OES service

‒ changes do not affect services that already have been configured

‒ use one single LDAP group to manage redundant LDAP servers

‒ create a wildcard certificate for redundant LDAP servers “*.myCompany.com”

• LDAP configuration for individual OES services(LUM, iFolder, iPrint, DHCP, DNS, CIFS, NCS, NetStorage)

‒ configure redundant LDAP servers wherever possible

‒ always select the closest LDAP server that has the required information replicated in its eDirectory database


Linux User Management (1)Novell® Open Enterprise Server

• Clean up the tree before implementing LUM‒ remove / clean up old POSIX attributes

‒ make sure that uniqueID attribute is correct

‒ make sure user identities are unique

• Unix Configuration Object (UCO)‒ use one UCO per eDirectory tree

‒ create and configure UCO before the first OES server is introduced to the tree

‒ place high in the tree, i.e. in ou=LUM.ou=Services.o=<Org>

‒ adjust uamPosixGidNumberLastAssignedand uamPosixUidNumberLastAssigned

• Unix Workstation Object‒ place in the server context


Linux User Management (2)Novell® Open Enterprise Server

• Primary LUM Group

‒ make sure each LUM user is a member of only one LUM group if there are specific services allowed on LUM groups

• /etc/nam.conf

‒ configure LDAP server that has a replica of the LUM related object's

‒ always configure alternative-ldap-server-list

‒ execute namconfig -k to get certificates;requires eDirectory authentication

‒ important configuration settings:

‒ cache-only=yes

‒ case-sensitive=no

‒ convert-lowercase=yes

‒ persistent-search=no


OES Proxy UsersNovell® Open Enterprise Server

• Used to obtain information from eDirectory™ on behalf of an OES service

• Place them in the same context as the server providing the service

• Use OES Common Proxy User !!‒ introduced with OES2 SP3

‒ execute /opt/novell/proxymgmt/bin/..../move_to_common_proxy.sh if upgradingfrom OES2 SP2 or earlier

‒ one OES proxy user for all OES servicesprovided by a server

‒ automated password management through cron

‒ security


OES Services (NCS)Novell® Open Enterprise Server

• Novell® Cluster Services (NCS)‒ have all related objects in the same organizational unit

‒ partition and replicate this OU to all cluster nodes

‒ configure to use local LDAP server

‒ use “/opt/novell/ncs/bin/ncs-configd.py -init”to verify configuration (“NCS sanity check”)

‒ configure and activate resource monitoring

‒ disable cascading failover prevention after implementing a failover matrix


OES Services (iPrint) (1)Novell® Open Enterprise Server

• Driver Store / Manager‒ place in server or cluster context

‒ use DNS Name / CNAME

‒ configure multiple eDirectory Servers

• Specific SSL certificates for iPrint ‒ create a SSL certificate for the secondary address

in eDirectory, export and convert it (pfx to pem).

‒ put the certificate in /etc/ssl/servercerts

‒ create a vhost-ssl-<iPrint-Service>.confwith VirtualHost <secondary-address:443>and SSLCertificateKeyFile /etc/ssl/servercerts/<secondary-certificate>.pem


OES Services (iPrint) (2)Novell® Open Enterprise Server

• Configure Apache fault tolerant and do not dereference alias objects

‒ /etc/opt/novell/iprint/httpd/conf/iprint_ssl.conf

AuthLDAPDNURL "ldaps://server1.de server2.de server3.de:636/???(objectClass=user)"AuthLDAPDNDereferenceAliases never

‒ /etc/openldap/ldap.conf

TLS_REQCERT never

• Replicate required eDirectory objects to the server hosting the service

‒ Manager and Printer Agents to the server hosting the manager

‒ DriverStore objects to the server hosting the DriverStore


OES Services (NSS)Novell® Open Enterprise Server

• Read ahead default‒ NW = 2 / OES = 16

‒ modify depending on your needs (i.e. to 64)

• nsscon /idcachesize=131072‒ increase depending on the number of trustees

• /etc/opt/novell/nss/nssstart.cfg‒ noatime depending on your backup solution

‒ activate XATTR for POSIX based backup and tools


OES Services (NCP)Novell® Open Enterprise Server

• ndsconfig set ‒ n4u.server.max-threads=[new value]

‒ new default value is 256

‒ monitor with ncpcon threads command

• ncpcon set (see also TID 7004888)‒ MAXIMUM_CACHED_FILES_PER_SUBDIRECTORY=10240

‒ MAXIMUM_CACHED_FILES_PER_VOLUME=256000

‒ MAXIMUM_CACHED_SUBDIRECTORIES_PER_VOLUME=102400

‒ LOCAL_CODE_PAGE=[your codepage]

‒ CONCURRENT_ASYNC_REQUESTS=50

‒ ADDITIONAL_SSG_THREADS =50

‒ new parameter NCP_TCP_KEEPALIVE_INTERVAL(default = 8 minutes; range 3 minutes to 240 minutes)

‒ do not modify FIRST_WATCHDOG_PACKETparameter unless instructed to do so by NTS


OES Services (CIFS)Novell® Open Enterprise Server

• context file vs. subtree search

• novcifs -o‒ Maximum Cached Subdirectories Per Volume - 102400

‒ Maximum Cached Files Per Subdirectory - 10240

‒ Maximum Cached Files Per Volume - 256000

‒ Oplocks - Enabled

‒ DFS - Enabled

‒ Cross Protocol Lock - Enabled

‒ Subtree Search - Disabled / Enabled

‒ Offline caching support at client - 0 / 1

‒ Block invalid users from authenticating - Enabled (Timeout period - 5 mins)

• ndsd restart dependency, rcnovell-cifs restart

• rcnovell-cifs monitor to crontab

Novell® Open Enterprise ServerAdministration


Administration (1)Novell® Open Enterprise Server

• Always use the latest version of the tools shipped with the latest OES version installed at your environment (eg. DNS)

• NDSRepair for UNIX Menu Wrapper

‒ http://www.novell.com/communities/node/2282/ndsrepair-unix-menu-wrapper

• Graphical NDS repair (ndsgrepair; OES11 SP1)

• ConsoleOne®

‒ only valid to manage GroupWise® or ZENworks® 7

• iManager

‒ have multiple instances on central eDirectory™ servers

‒ needs to be installed on each NetStorage server (minimal plugins)

‒ ensure all instances have exactly the same plugins

http://www.novell.com/communities/node/2282/ndsrepair-unix-menu-wrapper

http://www.novell.com/communities/node/2282/ndsrepair-unix-menu-wrapper


Administration (2)Novell® Open Enterprise Server

• NSSMU‒ frontend to NLVM library

‒ main storage administration tools

• Novell Linux Volume Manager command line interface‒ NLVM is new to OES11

‒ CLI supports all commands to mange storage

• NSSRAID‒ management utility for software RAIDs

‒ nlvm cli aliases

Novell® Open Enterprise ServerTroubleshooting


Troubleshooting (1)Novell® Open Enterprise Server

• Always obtain a supportconfig‒ see http://www.novell.com/communities/node/2332

‒ download the latest version from http://en.opensuse.org/Supportutils

‒ install appropriate plugins (Novell® GroupWise®, iPrint, NCS, etc.)

• TCPDUMP‒ quick LAN traces on the server

‒ tcpdump -s 0 -i <interface> -w <tracefile> interface: eth0, bond0, any, ...

‒ tcpdump -s 0 -i <interface> host x.x.x.x -w <tracefile>

‒ tcpdump -s 0 -i <interface> ip proto 224 -w <tracefile>

http://www.novell.com/communities/node/2332

http://en.opensuse.org/Supportutils



• Make your own directory for your temporary files

• How to upload via FTP from the server

#ftp ftp.novell.com

Name (ftp.novell.com:root): anonymous

Password: <Email address>

ftp> hash

ftp> bin

ftp> cd incoming

ftp> put <your file to upload>



• Display DEV_LOSS_TMO of all devices‒ for i in $(find /sys -iname dev_loss*);do echo $i; cat $i;done

• Top in batch mode‒ top -p `pid of <processname>` -d 600 -b > /TEMP/top.log

• Loop for logging(this example runs some simple I/O performance measurement)

#!/bin/bashfor (( ; ; )); do/bin/echo "-------START IOSTATS TESTING-------" >>/TEMP/iostatoutput.log/bin/date >>/TEMP/iostatoutput.log/bin/dd if=/dev/zero of=/var/opt/novell/eDirectory/data/dib/iotest.log bs=64k count=8k conv=fdatasync >>/TEMP/iostatoutput.log 2>&1sleep 30done



• STRACE & GSTACK * Troubleshooting Linux‒ strace -f -o <strace.log> -p <pid> > /TEMP/<filename-x.log> 1

‒ -f -- follow forks, -ff -- with output into separate files

‒ -p pid -- trace process with process id PID, may be repeated

‒ do several strace over a short time line to document the issues

‒ gstack <process-id> >/root/<filename-x.log>



• SLP Tool‒ ex. "slp.sh 10.65.17.144"

‒ #!/bin/bashfor i in $(slptool unicastfindsrvtypes $1);do slptool unicastfindsrvs $1 $i;done

‒ IMPORTANT: you have to use <unicast> - otherwise slptool will use broadcast/multicast for finding the services and will not show the "DA-Registrations", only.

• SCREEN‒ open session:

screen -S <screen name>

‒ attach other or reattach:screen -x <screen name>


Troubleshooting - Cores (1)Novell® Open Enterprise Server

• Kernel crash dump‒ dumps the state of the complete system

(NetWare: kernel core dump)

‒ kdump, kexec-tools and makedumpfile(note: kernel-kdump is not required on SLES11)

‒ configure according to TID 3374462

‒ reserve memory for the capture kernel

‒ activate kdump system service (chkconfig)

‒ configure kdump with YaST (filtering, autodelete, path)

‒ dumps go into /var/crash

• Always upload cores together with a supportconfig


Troubleshooting - Cores (2)Novell® Open Enterprise Server

• Application cores (core dump for a process)‒ Configure according to TID 3054866

‒ disable the limit for the maximum size of a core dump file(set SOFTCORELIMIT="unlimited" in /etc/sysconfig/ulimit)

‒ configure a fixed location for storing core dumps(install -m 1777 -d /var/local/dumps)

‒ configure core naming pattern(echo"/var/local/dumps/core.%e.%p"> /proc/sys/kernel/core_pattern)

%e = process name, %p = process ID

‒ Never put crash dumps and application cores in the same directory

‒ Always process core with novell-getcore / getappcore‒ General - TID 7004526

‒ eDirectory - TID 3078409

‒ GroupWise - TID 3447847

Question and Answer


Useful Links

• OES 11 Consulting Best Practice Guide‒ http://www.novell.com/documentation/oes11/mgmt_bp_guide

_lx/data/bookinfo.html

• The Novell Consulting Installation Framework - AutoYaST

‒ https://www.novell.com/communities/node/14216/novell-consulting-installation-framework-autoyast

http://www.novell.com/documentation/oes11/mgmt_bp_guide_lx/data/bookinfo.html

http://www.novell.com/documentation/oes11/mgmt_bp_guide_lx/data/bookinfo.html

https://www.novell.com/communities/node/14216/novell-consulting-installation-framework-autoyast

https://www.novell.com/communities/node/14216/novell-consulting-installation-framework-autoyast


Corporate Headquarters1800 South, Novell PlaceProvo, Utah 84606

801.861.7000 (Worldwide)800.453.1267 (Toll-free)

Join us on:www.novell.com

This document could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein. These changes may be incorporated in new editions of this document. Novell, Inc. may make improvements in or changes to the software described in this document at any time.

Copyright © 2011 Novell, Inc. All rights reserved.

All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States. All third-party trademarks are the property of their respective owners.

Technology

GWAVACon 2013: Novell Open Enterprise Server Best Practices