Upload
hoangliem
View
248
Download
2
Embed Size (px)
Citation preview
Delivering the Moment
SelenioNext™ Dense Multiscreen Transcoding Broadcast Management System (BMS)
Software Version 1.0 July 2013 Edition B 175-100509-00
Installation and Troubleshooting Guide
© 2014 Imagine Communications Corp. Proprietary and Confidential
Publication Information © 2014 Imagine Communications Corp. Proprietary and Confidential.
Imagine Communications considers this document and its contents to be proprietary and confidential. Except for making a reasonable number of copies for your own internal use, you may not reproduce this publication, or any part thereof, in any form, by any method, for any purpose, or in any language other than English without the written consent of Imagine Communications. All others uses are illegal.
This publication is designed to assist in the use of the product as it exists on the date of publication of this manual, and may not reflect the product at the current time or an unknown time in the future. This publication does not in any way warrant description accuracy or guarantee the use for the product to which it refers.
Imagine Communications reserves the right, without notice to make such changes in equipment, design, specifications, components, or documentation as progress may warrant to improve the performance of the product.
Trademarks Selenio™ and SelenioNext™ are trademarks of Imagine Communications or its subsidiaries. Microsoft® and Windows® are registered trademarks of Microsoft Corporation. All other trademarks and trade names are the property of their respective companies.
Contact Information Imagine Communications has office locations around the world. For locations and contact information see: http://www.imaginecommunications.com/contact-us/
Support Contact Information For support contact information see:
Support Contacts: http://www.imaginecommunications.com/services/technical-support/
eCustomer Portal: http://support.imaginecommunications.com
SelenioNext™
Dense Multiscreen Transcoding
Installation and Troubleshooting Guide
Software Version 1.0
July 2013 Edition B: June 2014
About this Document
SelenioNext Broadcast Management System V.1.0 2 Installation and Troubleshooting Guide
Publication Information
© 2014 Imagine Communications Corp. Proprietary and Confidential.
Imagine Communications considers this document and its contents to be proprietary and confidential. Except for making a reasonable number of copies for your own internal use, you may not reproduce this publication, or any part thereof, in any form, by any method, for any purpose, or in any language other than English without the written consent of Imagine Communications. All others uses are illegal.
This publication is designed to assist in the use of the product as it exists on the date of publication of this manual, and may not reflect the product at the current time or an unknown time in the future. This publication does not in any way warrant description accuracy or guarantee the use for the product to which it refers.
Imagine Communications reserves the right, without notice to make such changes in equipment, design, specifications, components, or documentation as progress may warrant to improve the performance of the product.
Trademarks
Selenio™ and SelenioNext™ are trademarks of Imagine Communications or its subsidiaries. Microsoft® and Windows® are registered trademarks of Microsoft Corporation. All other trademarks and trade names are the property of their respective companies.
Contact Information
Imagine Communications has office locations around the world. For locations and contact information see: http://www.imaginecommunications.com/contact-us/
Support Contact Information
For support contact information see:
Support Contacts: http://www.imaginecommunications.com/services/technical-support/
eCustomer Portal: http://support.imaginecommunications.com
About this Document
SelenioNext 3 Installation and Troubleshooting Guide
About this Document This document contains instructions for installing the software components of SelenioNext (processor, packager, and BMS) as well as guidelines for troubleshooting. Chapters and Their Contents
1 Introduction Key components and acronyms; Contact information
Pg. 6
2 Prerequisites and General Guidelines
Summary of connectors; guidelines and constraints
Pg. 8
3 Hardware Setup Setup of the Dell R720 2 RU server Pg. 9
4 Software Installation Basic installation procedure Pg. 18
5 BMS Configuration Setting essential BMS parameters and running the configuration command
Pg. 20
6 SelenioNext Setup Initial and repeat installations of Static or HA servers
Pg. 26
7 Post-Installation Procedures Verifying NTP, SNMP Support, Time Zones; Database Replication
Pg. 32
8 Troubleshooting Tests, log files and high-availability status Pg. 45
9 Maintenance Procedures Logins, access, errors, resets, shutdown, and reboot
Pg. 48
A Appendix: Creation of a System Definition File
Creation of a System Definition File Pg. 50
Style Conventions
Convention Used for
Verdana Regular text
Arial Bold Names of commands and other elements of the user interface
Arial Italics Special terms, the first time they appear
Monospace Text entered by a user, values of parameters, and usage examples
Notes, which offer an additional explanation or a hint on how to overcome a common problem
Warnings of potential problems
Table of Contents
SelenioNext Broadcast Management System V.1.0 4 Installation and Troubleshooting Guide
Table of Contents ABOUT THIS DOCUMENT ............................................................................. 3
TABLE OF CONTENTS .................................................................................. 4
1 INTRODUCTION .................................................................................... 6
1.1 ACRONYMS .........................................................................................6
2 PREREQUISITES AND GENERAL GUIDELINES........................................ 8
2.1 INSTALLATION FLOW ..............................................................................8
3 HARDWARE SETUP ................................................................................ 9
3.1 DELL R720 2 RU SERVER .......................................................................9
Installing the Next Cards in the Dell R720 Server ..................................9
Configuring the Dell R720 Server ...................................................... 16
3.2 EXTERNAL POWER FOR THE NEXT CARDS ..................................................... 16
4 SOFTWARE INSTALLATION ................................................................. 18
4.1 NEXT SOFTWARE INSTALLATION .............................................................. 18
4.2 BMS SOFTWARE INSTALLATION .............................................................. 19
5 BMS CONFIGURATION ........................................................................ 20
5.1 SETTING THE BMS IP AND GATEWAY ........................................................ 20
5.2 SETTING THE BMS HOSTNAME ................................................................ 20
5.3 RUNNING BMS-SETUP ......................................................................... 21
6 NEXT SETUP ........................................................................................ 26
6.1 INITIAL SETUP OF A STATIC (NON-HA) NEXT SERVER .................................... 26
Recovering Ethernet Interfaces ......................................................... 28
Initial Setup of IBS HA Cluster .......................................................... 28
6.2 REPEATED SETUP OF A SERVER (STATIC OR HA) ........................................... 29
7 POST-INSTALLATION PROCEDURES .................................................... 32
7.1 VERIFYING NTP ................................................................................. 32
7.2 SNMP SUPPORT ................................................................................ 33
7.3 MYSTRO MIB AND SELENIONEXT SNMP AGENT SUPPORT ............................... 34
7.4 TIME ZONE CHANGE ............................................................................ 35
7.5 ADDITIONAL BMS SETTINGS .................................................................. 36
7.6 SETTING UP DATABASE REPLICATION FOR BMS ............................................ 38
Master Configuration ....................................................................... 39
Table of Contents
SelenioNext 5 Installation and Troubleshooting Guide
Slave Configuration ......................................................................... 41
Recovery from a BMS Server Hardware Failure (Hot Standby) .............. 42
Recovery from a BMS Server Hardware Failure (Cold Standby) ............. 43
8 TROUBLESHOOTING ............................................................................ 45
8.1 VERIFYING NEXT OUTPUT STREAM ............................................................ 45
8.2 VERIFYING VIDEO SOURCES ................................................................... 45
8.3 TESTING SELENIONEXT EQUIPMENT OPERATION ........................................... 46
8.4 CHECKING LOG FILES .......................................................................... 46
8.5 VERIFYING STATUS OF HIGH AVAILABILITY .................................................. 46
9 MAINTENANCE PROCEDURES .............................................................. 48
9.1 LOGGING INTO A SERVER ...................................................................... 48
9.2 SHUTTING DOWN A SERVER ................................................................... 48
9.3 REBOOTING A SERVER .......................................................................... 48
APPENDIX A – CREATION OF A SYSTEM DEFINITION FILE ....................... 50
Introduction
SelenioNext Broadcast Management System V.1.0 6 Installation and Troubleshooting Guide
1 Introduction
SelenioNext™ makes it possible for video consumer to access programming across an array of mobile and fixed devices and networks. For service providers, SelenioNext maximizes the quality, quantity, and footprint required in a multiscreen, TV Everywhere environment.
SelenioNext is an all-in-one TV Everywhere solution that replaces an entire headend of video processing in a single commercial-off-the-shelf (COTS) server platform. The SelenioNext solution is Enabling PersonalizedTV® by allowing service providers to ingest precompressed services, and then transcode, package, encrypt and stream multiscreen, multi-device video. Designed to provide live programming to mobile and connected screens, SelenioNext delivers a PersonalizedTV® experience, enabling video consumers to view and interact with content on the device of their choice at any time or location.
SelenioNext is available in 1RU, 2RU, or 10RU blade systems that fit into optimal form factors for all online video applications—from providing several IPTV streams, to thousands of multiscreen transcodes. Using advanced Adaptive Bit Rate (ABR) technology, the system supports up to 320 HD ABR or 320 SD ABR profiles per 2RU server, and scales up to any number of profiles per video program. SelenioNext also features an onboard broadcast management system for control and visibility into network resource optimization.
The SelenioNext software package holds several key components that are introduced in this document. A SelenioNext processor creates multiple output SPTS streams and multiple ABR groups. Additionally, it can package those streams in various formats.
The key components of SelenioNext can be installed in any sequence. However, several prerequisites must be met before installation: see Chapter 2.
1.1 Acronyms
You should be familiar with the following terms when installing the system.
Acronym Meaning
ICE Interchangeable Compressed Elements
IBS ICE Broadcast System
BMS Broadcast Management System
iLO Integrated Lights-Out Management (HP c7000)
AVP Advanced Video Processor
Introduction
SelenioNext 7 Installation and Troubleshooting Guide
MIB Management Information Base
SNMP Simple Network Management Protocol
NTP Network Time Protocol
Prerequisites and General Guidelines
SelenioNext Broadcast Management System V.1.0 8 Installation and Troubleshooting Guide
2 Prerequisites and General Guidelines
As a prerequisite to the installation process, management network interfaces must be configured prior to installation of SelenioNext software.
SelenioNext version 1.0 works on Linux v3.0 (internal release number).
2.1 Installation Flow
The installation process includes the following major tasks:
1. Install hardware, including Linux, on the hardware platform:
Dell PowerEdge R720 (see Section 3.1 Dell R720 2 RU Server on page 9)
2. Install SelenioNext software (see Section 4.1 SelenioNext Software Installation on page 18).
3. Install BMS software (see Section 4.2 BMS Software Installation on page 19).
4. Configure BMS (see Chapter 5 BMS Configuration on page 20).
5. Set up SelenioNext (see Chapter 6 SelenioNext Setup on page 26).
Hardware Setup
SelenioNext 9 Installation and Troubleshooting Guide
3 Hardware Setup
3.1 Dell R720 2 RU Server
Installing the SelenioNext Cards in the Dell R720 Server
You can install up to two SelenioNext cards in each of the left and right chambers (orientation when looking from the back side) of the Dell R720 server, as described in the procedures in the following sections. If you want to install the maximum allowable four SelenioNext cards, you should start the installation from the left chamber.
Ensure that all power is removed from the server before removing or installing cards. You must disconnect the power cables before performing this procedure.
Perform electrostatic discharge and antistatic prevention measures prior to handling the server.
Installing SelenioNext cards in the left chamber
To install the SelenioNext cards in the left chamber of the Dell R720 server:
1. After removing the server cover, disconnect and remove riser 3.
Removing Riser 3
2. Make sure the two light blue card fasteners are pulled out.
Hardware Setup
SelenioNext Broadcast Management System V.1.0 10 Installation and Troubleshooting Guide
Figure 1 – Card Fasteners Pulled Out
3. Connect the power cable to riser 2.
Figure 2 – Connecting Power Cable to Riser 2
4. Connect the power cable to the first card, and then insert the card into the bottom slot of riser 2.
Hardware Setup
SelenioNext 11 Installation and Troubleshooting Guide
Figure 3 – Connecting First Card
5. Insert the second card into the top slot of riser 2 without connecting the power cable to the card.
Figure 4 - Installing Second Card
6. Connect the power cable to the second card.
Hardware Setup
SelenioNext Broadcast Management System V.1.0 12 Installation and Troubleshooting Guide
Figure 5 – Connecting Power Cable to Second Card
7. Lock the two light blue card fasteners that were pulled out in step 2.
Figure 6 – Locking Cards into Place
8. Reconnect riser 3 to the motherboard.
Hardware Setup
SelenioNext 13 Installation and Troubleshooting Guide
Figure 7 – Reconnecting Riser 3 to Motherboard
9. Lock the left chamber cards by clicking the blue button.
Figure 8 – Locking Cards in Left Chamber
Installing SelenioNext cards in the right chamber
To install the SelenioNext cards in the right chamber of the Dell R720 server:
1. Make sure the two light blue card fasteners are pulled out, and connect the power cable to riser 3.
Hardware Setup
SelenioNext Broadcast Management System V.1.0 14 Installation and Troubleshooting Guide
Figure 9 – Connecting Power Cable to Riser 3
2. After connecting the power cable to the first card, insert the card into the bottom slot of riser 3.
Figure 10 – Connecting First Card
3. After connecting the power cable to the second card, insert the card into the top slot of riser 3.
Hardware Setup
SelenioNext 15 Installation and Troubleshooting Guide
Figure 11 - Connecting Second Card
4. Lock the two light blue card fasteners that were pulled out in step 1.
Figure 12 – Locking Cards into Place
5. Lock the left chamber cards by clicking the blue button.
Hardware Setup
SelenioNext Broadcast Management System V.1.0 16 Installation and Troubleshooting Guide
Figure 13 – Locking Cards in Right Chamber
Configuring the Dell R720 Server
To configure the Dell R720 server:
1. Boot into BIOS (press DEL when prompted).
2. Make sure the settings are the same as listed below, and save and exit at the end of the process.
3. In iDRAC Settings, in Thermal menu, change Thermal Base Algorithm to Maximum performance, and change Fan Speed offset to High…
4. Under Miscellaneous Settings, disable F1/F2 Prompt on Error, and set Keyboard Error to Do Not Report.
3.2 External power for the SelenioNext cards
The SelenioNext cards need to receive external power through a 6-pin PCI Express power cable. The connector on the board can be seen in the following figure; only one of the two cards needs to be connected:
Hardware Setup
SelenioNext 17 Installation and Troubleshooting Guide
Figure 14 - Connectors
The Dell R720 server requires a Y cable for each pair of SelenioNext cards.
More information can be obtained in the following site:
http://www.playtool.com/pages/psuconnectors/connectors.html#pciexpress
Software Installation
SelenioNext Broadcast Management System V.1.0 18 Installation and Troubleshooting Guide
4 Software Installation
Before starting this section please make sure you have the latest version of Linux installed, along with all the latest dependencies.
Software installation includes installation of SelenioNext (including co-located packager) and BMS.
4.1 SelenioNext Software Installation
The installation of Next version 1.0 is divided into several parts; the various software components are divided into three RPM packages in total (where XX is the most recent build number):
next-1.0.0-182.x86_64.rpm: contains major binaries and setup files
next-common-1.0.0-182.x86_64.rpm: contains common binaries and files
bms-next-1.0.0-182.x86_64.rpm: contains BMS application server files and scripts
In order to install Next v1.0 for the first time, please perform the server installation from scratch using the Tarball file. For further instructions refer to chapter 11 of the SelenioNext Configuration Manual.
In case of an upgrade, place all three packages in a single folder and issue the following command from that folder (using root user):
Rpm –Uvh *.rpm (installing all RPMs that are located in the folder)
Software Installation
SelenioNext 19 Installation and Troubleshooting Guide
4.2 BMS Software Installation
BMS installation is separated from SelenioNext SW installation and requires only one package.
Copy RPM into destination folder and issue:
rpm –Uvh bms-next-1.0.0-182.x86_64.rpm
During the installation process, you are provided with valuable information such as server vendor, product name and Bios version. These help avoid installation/racking problems.
BMS Configuration
SelenioNext Broadcast Management System V.1.0 20 Installation and Troubleshooting Guide
5 BMS Configuration
Configuration of BMS involves:
Setting the BMS IP and Gateway
Setting the BMS Hostname
Running BMS-setup
5.1 Setting the BMS IP and Gateway
Open the file /etc/sysconfig/network-scripts/ifcfg-eth2, and verify the following settings:
IPADDR=a.b.c.d
NETMASK=255.255.x.x
GATEWAY=x.x.x.x
Onboot=yes
BOOTPROTO=static
If the server is not connected to the data network it is recommended to edit the file /etc/sysconfig/network-scripts/ifcfg-eth0, changing this line:
Onboot=yes
to
Onboot=no
5.2 Setting the BMS Hostname
Open the file /etc/sysconfig/network, and change the following setting:
HOSTNAME=<name>
HOSTNAME must start with an alphabetical letter. For example: Server-100 is legal, while 103_20 is illegal.
The hostname is valid only after a reboot command is issued to the server (recommended).
Hostnames containing the underscore character (“_”) are not supported.
BMS Configuration
SelenioNext 21 Installation and Troubleshooting Guide
5.3 Running BMS-setup
Use the following script for first-time installation as well as for upgrades.
All binaries and scripts are already in your PATH; there is no need to change the PATH to look for them.
To complete BMS configuration:
1. Start BMS setup by typing:
bms-setup.sh
You are prompted for the BMS hostname, BMS IP address, and NTP IP.
2. Press <Enter> for the default BMS hostname (206-50 in the example above) and press <Enter> once again for the default IP address of the BMS. Enter the NTP IP or press <Enter> to use the current default.
BMS hostname must be resolvable to assure proper operation of the BMS server.
The following screen appears.
BMS Configuration
SelenioNext Broadcast Management System V.1.0 22 Installation and Troubleshooting Guide
3. To the question: "Is this an all-in-one configuration (currently no)? [y/N)", type the response y if the BMS and IBS are running on the same server. Otherwise, press <Enter> (for the default answer No).
During the bms-setup run, you are asked whether you want to wipe out the database. Do so only as part of a new installation on a machine. If you do so in any other case (for example, between upgrades) all saved information will be lost.
4. bms-setup.sh first resolves the server hostname. Afterwards, the user is prompted to specify whether the system should set up a SECTAMUS trap aggregator.
BMS Configuration
SelenioNext 23 Installation and Troubleshooting Guide
5. The BMS can aggregate all traps and send them to a user-defined IP. This IP must be resolvable. This adds an entry in the format “x.x.x.x compass” to /etc/hosts.
BMS Configuration
SelenioNext Broadcast Management System V.1.0 24 Installation and Troubleshooting Guide
After adding trap-aggregator IP(s), the following question appears:
"Do you want to start the BMS services right now? (may take a while) [Y,n]:"
6. Press <Enter> to start the processes, and wait until the status “Finished” appears. (This takes a few minutes.)
When the process is complete, the following message appears:
"IBS Management setup is now complete"
BMS Configuration
SelenioNext 25 Installation and Troubleshooting Guide
SelenioNext Setup
SelenioNext Broadcast Management System V.1.0 26 Installation and Troubleshooting Guide
6 SelenioNext Setup
Procedures for both initial and subsequent setup are described below.
6.1 Initial Setup of a Static (non-HA) Next Server
To set up a static SelenioNext server for the first time:
1. Run ibs-setup. (Note that ibs-setup remembers the last user entries).
Select the host type from the 4 options (see screenshot above).
2. Enter data as requested during the setup process:
SelenioNext Setup
SelenioNext 27 Installation and Troubleshooting Guide
SelenioNext Setup
SelenioNext Broadcast Management System V.1.0 28 Installation and Troubleshooting Guide
The setup takes a few moments: Ethernet interfaces are listed, the route table is displayed, and the “finished running install script” notification appears. The user is returned to the command shell:
Defining a server’s role is done in the BMS, in the Auto Discovery dialog. Please refer to the SelenioNext BMS Configuration manual for more details.
Recovering Ethernet Interfaces
When running ibs-setup, the user is asked about restoring the interfaces with the user’s answers from previous runs of the command.
Initial Setup of IBS HA Cluster
The assumption is that the management stub IPs of all cluster machines are configured in advance. Please refer to Setting the BMS IP and Gateway on page 20.
SelenioNext Setup
SelenioNext 29 Installation and Troubleshooting Guide
To set up an HA cluster for the first time:
1. Create a system file describing the HA cluster, and place it on one of the machines in the cluster. For instructions on how to create the file, see Appendix A.
2. Run: ibs-setup-all --system <system file>
The /etc/hosts file on the machine running ibs-setup-all must be fully populated with data for all servers in the cluster.
3. When prompted, provide passwords for the qoduser and root accounts.
The script will configure the whole cluster, as defined by the settings file. During the process, you will be notified about each server being configured, and the status of the configuration.
6.2 Repeated Setup of a Server (Static or HA) During the setup process, a cache of the configuration is kept, and reconfiguring the server after a software upgrade is a straightforward procedure.
To repeat setup (e.g., after a software upgrade):
1. Run ibs-setup.
2. When asked whether to reuse (restore) the cached configuration of the server, press <Enter> to reply Yes.
SelenioNext Setup
SelenioNext Broadcast Management System V.1.0 30 Installation and Troubleshooting Guide
SelenioNext Setup
SelenioNext 31 Installation and Troubleshooting Guide
If you do not wish to reuse the configuration, type anything other than yes or y, press <Enter>, and proceed to specify the various configuration parameters as described earlier.
When installation is complete, the “finished running install script” message appears:
If a network interface modification is needed, and reuse of the cache was specified, ibs-setup will fail. In order to force the network modifications, type the following: ibs-setup –r –-modify-net If a setup is needed without restarting network service, type the following: ibs-setup –rn
Post-Installation Procedures
SelenioNext Broadcast Management System V.1.0 32 Installation and Troubleshooting Guide
7 Post-Installation Procedures
The following procedures can be done after IBS has been installed:
Verifying NTP
SNMP Support
Mystro MIB and SelenioNext SNMP Agent Support
Time Zone Change
Additional BMS Settings
Setting Up Database Replication for BMS
7.1 Verifying NTP
Issue the following command (after running ibsinstall) to check that the NTP is working properly.
root@host etc]# ntpq –p remote refid st t when poll reach delay offset jitter ============================================================================== *imagine-ntp.ima 192.114.62.249 3 u 3 64 377 0.093 1.044 0.234 LOCAL(0) .LOCL. 10 l 4 64 377 0.000 0.000 0.001
If imagine-ntp cannot be found, do the following:
1. Place the following line in /etc/hosts:
xxx.xxx.xxx.xxx imagine-ntp
2. Make sure that the IP is pingable.
To check that the NTP server is functional:
1. Stop NTPD by typing:
service ntpd stop
2. Issue the command:
ntpdate imagine-ntp
If this command is successful, a message such as the following appears:
Post-Installation Procedures
SelenioNext 33 Installation and Troubleshooting Guide
28 Apr 07:36:36 ntpdate[17307]: adjust time server xxx.xxx.xxx.xxx offset -0.010892 sec
Otherwise, one of the following errors may appear:
28 Apr 07:37:18 ntpdate[17313]: no server suitable for synchronization found
(The server is pingable but not an NTP server.)
OR
Error: Name or service not known
28 Apr 07:37:04 ntpdate[17310]: can't find host imagine-ntp2
(The server is not pingable.)
3. Once done, type the following command to restart NTP:
service ntpd start
7.2 SNMP Support
SelenioNext software support sends SNMP Trap notifications to designated places of your choice. The SelenioNext processor looks for a special file that contains the IP addresses of all SNMP Trap listeners that you chose.
Post-Installation Procedures
SelenioNext Broadcast Management System V.1.0 34 Installation and Troubleshooting Guide
The filename is:
/etc/ibs/receivers
and the format is straightforward:
<ip-address>[:port]
For example:
172.16.0.140 192.168.10.165:1660 10.1.1.1:30000
Upon startup, the qod-mcs and qod-ivp read this file and send a trap to:
172.16.0.140 (using the default port: 162)
192.168.10.165 (using port 1660)
10.1.1.1 (using port 30000)
If the file does not exist, traps will not be configured, until configuration is done using either the qodcli utility (for a standby server in an HA cluster) or BMS (for all other servers).
You are advised to edit the receivers file of the standby server before installation. If you edit this file after the installation is complete, you must restart the relevant process. The BMS server’s IP is automatically added to the receivers list.
Please copy the SelenioNext MIB file that can be found at /usr/share/ibs/snmp/mibs/IMAGINE-MIB.my to your SNMP trap receiver.
7.3 Mystro MIB and SelenioNext SNMP Agent Support
To enable Mystro MIB support:
1. If this is the first time Mystro MIB support is installed, make sure this is a fresh install. This applies to both BMS and SelenioNext processor. The BMS DB is maintained; however, it is recommended that you export it as an extra precaution.
2. After installation, if necessary, re-configure qod.conf in the SelenioNext server (for VLANID tagging).
3. In /usr/share/ibs/PRM/conf/prm.properties on the SelenioNext processor, set “programTraps.twc=on”. This requires a PRM restart; therefore, do this before ibs-setup-host.
Post-Installation Procedures
SelenioNext 35 Installation and Troubleshooting Guide
There is no need to configure the trap receiver in the BMS GUI or IP forwarding on the BMS.
4. Configure the Mystro server in BMS snmptrapd.conf – “forward .1.3.6.1.4.1.14373 SECTAMUS:161”
5. snmpd and snmptrapd need to be running on BMS; snmpd needs to be running on the SelenioNext processor.
Additional settings to support Mystro MIB are located in the section Additional BMS Settings.
To enable SelenioNext SNMP agent, refer to the section Additional BMS Settings. This will enable the operator to perform the following commands:
Failover next: processor #n to standby
Fail primary input – forcing secondary input
Mux server reconcile
Output TS reconcile
Output program reconcile
IBP server reconcile
7.4 Time Zone Change
The Linux running on SelenioNext (all versions) time zone is defaulted to the US/Eastern time zone (i.e., New York). If you wish to change this, do the following (for example):
[root@host root]# ln –s /usr/share/zoneinfo/US/Mountain /etc/localtime
This will set the time zone of the machine to Mountain time.
The U.S. spans 12 time zones:
Alaska
Aleutian
Arizona
Central
Eastern
East-Indiana
Post-Installation Procedures
SelenioNext Broadcast Management System V.1.0 36 Installation and Troubleshooting Guide
Hawaii
Indiana-Starke
Michigan
Mountain
Pacific
Samoa
7.5 Additional BMS Settings
The following properties should be added/updated in the bms.properties file on the BMS machine (located under $JBOSS_HOME/server/default/conf).
The values of the properties can be adjusted according to the system requirements; however, ranges should be configured properly.
Note: changing any of the parameters listed below requires the creation of a new database.
USE_IMP_SSM=true
This parameter defines the method of IP allocation for IMP messages.
If the flag is set to false, allocate a unique IP and port per output TS.
If the flag is set to true, allocate a unique IP per output program/output ghost, and a unique port per output TS.
IMP_IP_POOL_NUMBER=1
This parameter determines whether one or two IMP pools will be used. Default value is 1. If the value is set to 2, two IMP pools are used; one for main mux and one for protect mux (in this case, ALLOCATED_IP_PROTECT must be defined).
ALLOCATED_IP_MIN_RANGE=230.1.1.1 and
ALLOCATED_IP_MAX_RANGE=230.1.2.50
This is the range of multicast IP addresses defined for multicast IMP messages. If IMP_IP_POOL_NUMBER=2, this range will be used for the main mux. If IMP_IP_POOL_NUMBER=1 or not defined, this range will be used for all multicast IMP messages.
ALLOCATED_IP_PROTECT_MIN_RANGE=233.1.1.1 and
Post-Installation Procedures
SelenioNext 37 Installation and Troubleshooting Guide
ALLOCATED_IP_PROTECT_MAX_RANGE=233.1.2.50
This is the second range of multicast IP addresses defined for multicast IMP messages. Only if IMP_IP_POOL_NUMBER=2 this range will be used for the protect mux.
ALLOCATED_PORT_MIN_RANGE=10000 and
ALLOCATED_PORT_MAX_RANGE=10300
This is the range of ports used for IMP and rate control messages.
ALLOCATED_LOCAL_AVP_MULTICAST_IP=230.100.1.0
This is the multicast IP defined for multicast IMP messages between the input and AVP.
SDI_SERVERS_IP=
This is a list (comma separated) of server IPs that have SDI inputs.
USE_FLAT_ORIGIN_HIERARCHY=false
This is a list (comma separated) of server IPs that have SDI inputs. If set to true, ICE streamer generates output files under the origin base URL, without folders.
Note: Running bms-setup –p allows the user to edit the following BMS properties: • IMP_IP_POOL_NUMBER • ALLOCATED_IP_MIN_RANGE • ALLOCATED_IP_MAX_RANGE • ALLOCATED_IP_PROTECT_MIN_RANGE • ALLOCATED_IP_PROTECT_MAX_RANGE • ALLOCATED_PORT_MIN_RANGE • ALLOCATED_PORT_MAX_RANGE
USER_NETWORK_ID=1
This is the original network ID field in IBS generated SDT.
PROGRAM_POLLING_TIME_DEFAULT=1000
PROGRAM_BITRATE_DIFF_THRESHOLD_DEFAULT=50
Input redundancy default parameters: polling time in milliseconds, threshold in %
EXPORT_FOLDER=/var/opt
Post-Installation Procedures
SelenioNext Broadcast Management System V.1.0 38 Installation and Troubleshooting Guide
The location on the BMS server where the exported BMS DB file is written
UPLOAD_FOLDER=/tmp
The location on the BMS server where the imported BMS DB file is written. This setting is used internally by BMS and should not be modified.
OUTPUT_TS_MAX_SCORE=16
Maximum resources on output TS
OUTPUT_TS_HD_SCORE = 4
The weight of an HD program in the output TS
OUTPUT_TS_SD_SCORE = 1
The weight of an SD program in the output TS
MYSTRO_SNMP_AGENT_SUPPORT = false
Change to true to support Mystro SNMP MIB.
IMAGINE_SNMP_AGENT_SUPPORT = false
Change to true to support SelenioNext SNMP MIB.
IMAGINE_SNMP_AGENT_READ_COMMUNITY = public
IMAGINE_SNMP_AGENT_WRITE_COMMUNITY = public
SNMP community
IMAGINE_SNMP_AGENT_SOURCE_FILTER =
List of requesting IPs that will be filtered; Use “,” as a delimiter
JBOSS must be restarted after changing parameters in the bms.properties file, and any existing sessions must be reconfigured. Also, a fresh copy of the bms.properties file is copied into the JBOSS on Fresh installation. In most cases, the properties are commented with a “#” to indicate that this property is already set in the code. For example: #OUTPUT_TS_MAX_SCORE=15 specifies a hard-coded value.
7.6 Setting Up Database Replication for BMS
It is important to continually replicate the database on the main BMS server, in order
Post-Installation Procedures
SelenioNext 39 Installation and Troubleshooting Guide
to sustain the system in case of hardware problems, server malfunctions or other uncertain situations.
The following step-by-step procedure describes how to make sure the database is fully replicated to another BMS server (backup/protect).
A number of issues to note:
It is assumed that you have used bms-setup.sh to configure a BMS server, and have the server running.
The secondary server must go through the entire configuration process using bms-setup.sh, without starting the processes.
During this time, do not perform any transactions using the BMS GUI screens.
The local properties file must be copied manually from the master to the slave.
If there is no spare BMS server on site, please back up the BMS DB after installation and setup, and manually copy the database files to a different server.
Master Configuration
Copy the /usr/local/imagine/mgmtservices/jboss/server/default/conf/local.properties file from the master server to the slave server.
To set up the master server:
1. Edit /etc/my.cnf: Under the [mysqld] section, add the following section:
binlog_format=mixed log-bin = mysql-bin binlog-do-db=jbossdb binlog-do-db=bms_db server-id=1 innodb_flush_log_at_trx_commit=1 sync_binlog=1 expire_logs_days=2 max_binlog_size=100M
2. Restart MySQL:
/etc/init.d/mysqld restart
3. Log into the MySQL database as root and grant the user 'root' replication privileges:
Post-Installation Procedures
SelenioNext Broadcast Management System V.1.0 40 Installation and Troubleshooting Guide
mysql -u root -p Enter password:
This brings you to the MySQL shell.
GRANT REPLICATION SLAVE ON *.* TO 'root'@'%' IDENTIFIED BY 'root'; FLUSH PRIVILEGES;
Note that in this example the username is 'root' and password is 'root'.
SHOW MASTER STATUS;
The last command will result in a display similar to this:
+---------------+----------+--------------+------------------+ | File | Position | Binlog_do_db | Binlog_ignore_db | +---------------+----------+--------------+------------------+ | mysql-bin.004 | 98 | bms_db,jbossdb | |+--------------+----------+--------------+------------------+ 1 row in set (0.00 sec)
Make a note of this information, as you will need it later on for the slave.
4. Leave the MySQL shell:
quit;
Post-Installation Procedures
SelenioNext 41 Installation and Troubleshooting Guide
5. Issue these commands:
mysqldump -u root -proot --opt jbossdb > jbossdb.sql mysqldump -u root -proot --opt bms_db > bms_db.sql
6. Copy these files into the slave server using your method of choice (for example, scp).
Slave Configuration
To set up the slave server:
1. If the mysqld daemon is not started, start it using the command:
service mysqld start
2. Use the following command to clean databases and ensure that the jboss database is ready to receive the table structure:
/usr/share/ibs/scripts/create_db.sh
3. Follow with these commands to copy the database structure from the master database into the slave:
mysql -u root -proot jbossdb </home/qoduser/jbossdb.sql mysql -u root -proot bms_db </home/qoduser/bms_db.sql
4. Make a copy of the /etc/my.cnf file by issuing the command:
cp /etc/my.cnf /etc/my.cnf.backup
5. Edit /etc/my.cnf to contain the following:
server-id=2 master-host=<master-ip-address> master-user=root master-password=root master-connect-retry=60 replicate-do-db=jbossdb replicate-do-db=bms_db slave-skip-errors=1062 expire_logs_days=2 max_binlog_size=100M
6. Restart MySQL:
/etc/init.d/mysqld restart
Post-Installation Procedures
SelenioNext Broadcast Management System V.1.0 42 Installation and Troubleshooting Guide
7. Enter the following:
mysql -u root -p Enter password: SLAVE STOP; Change MASTER to MASTER_HOST='<master-ip-address>', MASTER_USER='root', MASTER_PASSWORD='<root>', MASTER_LOG_FILE='mysql-bin.004', MASTER_LOG_POS=98;
Notice the master_log_file and master_log_pos. They appeared in the 'show master status;' command issued at the master server.
8. At the mysql prompt, enter:
START SLAVE; quit;
When jobss/jobss1 is updated on the master, all changes will be replicated to the same database on the slave.
Do not start the jboss daemon on the slave if the master is running. If you do so you will experience faults.
Recovery from a BMS Server Hardware Failure (Hot Standby)
If the Primary BMS server has malfunctioned, and the standby BMS server that has the replicated database is a BMS protect server, use the following switchover procedure.
To recover from a hardware failure (hot standby):
1. Shut down the Primary (main) BMS.
2. On the BMS protect server:
Edit /etc/sysconfig/network file, and change hostname to the main BMS name (for example CMC01-BMS_main)
Edit /etc/sysconfig/network-scripts/ifcfg-eth2 and change IP address to the main BMS IP
Type chkconfig jboss on.
Reboot.
At this point there will be a master BMS server and no ‘protect’ server.
Post-Installation Procedures
SelenioNext 43 Installation and Troubleshooting Guide
3. When the former ‘main’ BMS server is brought back online or replaced by a new server, prepare a ‘protect’ server as follows:
Install the latest BMS software (if necessary)
Run bms-setup.sh, choosing to do a FRESH installation for mysql DB.
Repeat the slave configuration procedure on this server.
4. Follow the master configuration procedure on the NEW ‘main’ BMS server.
The two servers have now switched roles. The former protect is now main, and vice versa.
Recovery from a BMS Server Hardware Failure (Cold Standby)
If the Primary BMS server has malfunctioned, the replicated database ensures a quick recovery process.
To recover from a hardware failure (cold standby):
Follow this procedure carefully in order to maintain data integrity. Be sure to do all steps as the root user.
1. Prepare a new BMS server, loading Linux v3.0 for SelenioNext unto it.
2. Specify a hostname for the server, and the same IP address that it had before the failure. This is highly important; otherwise, you will not be able to control the MUXes. Make sure the server is ready to function as a BMS server.
3. Run bms-setup.sh on the new server. Do not start the BMS server at this point: MySQL needs to be running in order to load the primary data.
4. On the BACKUP server, make sure there is no instance of jboss running. (If there is, issue the service jboss stop and wait to see if jboss has stopped by issuing: ps –ef | grep java. If nothing appears, then the jboss service is stopped and you may proceed to the next step.)
5. Enter the following commands:
mysqldump -u root -proot --opt jbossdb > jbossdb.sql mysqldump -u root -proot --opt bms_db > bms_db.sql
6. Copy these files onto the NEW primary BMS server.
7. On the Backup server, copy /etc/my.cnf.backup (the backup file you created earlier when you set up the mysql slave server) to /etc/my.cnf).
8. Restart mysqld:
Post-Installation Procedures
SelenioNext Broadcast Management System V.1.0 44 Installation and Troubleshooting Guide
server mysqld restart
At this point, the backup server is standalone.
9. On the primary BMS server, import the raw data you have copied from the backup server:
mysql -u root -proot jbossdb <jbossdb.sql mysql -u root -proot bms_db <bms_db.sql
10. Start the jboss server on the Primary server. You should be able to log in and work as usual.
It is important to be aware that currently there is NO REPLICATION between master and slave servers. In order to duplicate the above, you must repeat the master replication procedure.
Troubleshooting
SelenioNext 45 Installation and Troubleshooting Guide
8 Troubleshooting
If you are experiencing trouble, the following procedures should be followed:
Verifying SelenioNext Output Stream
Verifying Video Sources
Testing SelenioNext Equipment Operation
Checking Log Files
Verifying Status of High Availability
8.1 Verifying SelenioNext Output Stream
To verify a SelenioNext output stream:
1. Use an h.264 analyzer to check the SelenioNext output.
2. Verify that the bit rate is in accordance with the setup.
3. Verify that the configured program exists within the SPTS.
4. Verify for the configured program video bit rate and audio bit rate. AC-3 audio bit rate should be approximately 384 kbps.
5. Verify an SCTE-35 PID (if applicable).
8.2 Verifying Video Sources
To verify video sources:
1. Use an MPEG-2/h.264 analyzer to check the sources.
2. Verify an IP bit rate for the source.
3. Verify that the source program is there.
4. Verify for each program a video bit rate and audio bit rate.
5. Verify an SCTE-35 PID (if applicable).
6. Right-click on a stream and select Media Info.
You should see video and audio PIDs with green (OK) status.
7. If there are no video sources, verify correct functionality of the IP routers, input ASI->IP conversion device, and/or satellite receiver.
Troubleshooting
SelenioNext Broadcast Management System V.1.0 46 Installation and Troubleshooting Guide
8.3 Testing SelenioNext Equipment Operation
To test the SelenioNext equipment operation:
1. If the sources are good, and there is still no output from the SelenioNext (or programs are missing), check the SelenioNext servers.
2. If only one stream is missing from the output, check the IBP that is processing that stream:
Is the IBP powered on?
Are all Ethernet cables firmly connected? You can issue a ping to the server to check connectivity to the router.
Can you access the IBP via the SSH console?
3. If all streams are missing from the output of the ICE Mux, and the inputs are OK, check the ICE Mux.
Is the ICE Mux powered on?
Are the Ethernet cables firmly connected? You can issue a ping to the server to check connectivity to the router.
Can you access the ICE Mux via the SSH console?
(Note – the above procedure is using stat mux as an example, it is similar for CSP or any other SelenioNext application).
8.4 Checking Log Files
To check the log files:
1. SSH to the SelenioNext server.
2. Tail the log file with “tail –f /var/log/ibs/avp.log”.
3. Check if new events are being displayed in the log file. If no events are being logged, make sure that the Ethernet interface is up (by using the Linux utility ethtool eth0 (or eth2)), and that there are no problems with the Video source or router.
8.5 Verifying Status of High Availability
To check the status of High Availability:
1. SSH to one of the processors that is part of the cluster.
2. Issue the following command: sudo /usr/sbin/hare-mon -b (or –bx to include comments)
Troubleshooting
SelenioNext 47 Installation and Troubleshooting Guide
Verify that the username “qoduser” is located in the sudoers list before running this command. Otherwise, run as root.
3. Verify that all cluster hosts are present.
Maintenance Procedures
SelenioNext Broadcast Management System V.1.0 48 Installation and Troubleshooting Guide
9 Maintenance Procedures
The following sections detail the maintenance procedures for each of the following scenarios:
Logging into a Server
Shutting Down a Server
Rebooting a Server
9.1 Logging into a Server
Putty and secure CRT are both popular SSH clients on the Microsoft Windows OS.
Using putty, log into the BMS_Main server as follows:
1. If you are at a KVM connected to the BMS, open a console window.
2. Type ‘ssh qoduser@servername’, where servername is the name of the server. The password is qoduser.
3. If you need root access, issue ‘su –‘. The password is XXX.
9.2 Shutting Down a Server
This will be service affecting.
To shut down a server:
1. Log in to the server you want to shut down by following the procedure in Section 9.1 Logging into a Server.
2. Switch to the root user with: 'su -'. The root password is XXX.
3. Shut down the server with: init 0 or with poweroff.
The server shuts down gracefully.
9.3 Rebooting a Server
Use this procedure as an alternative to rebooting from the BMS screen: selecting the server and clicking the Reboot tool button.
This will be service affecting.
Maintenance Procedures
SelenioNext 49 Installation and Troubleshooting Guide
To reboot a server:
1. Log in to the server you want to shut down by following the procedure in Section 9.1 Logging into a Server.
2. Switch to the root user with: 'su -'. The root password is XXX.
3. Reboot the server with the command reboot.
The server reboots. Boot time should be approximately three to four minutes.
Maintenance Procedures
SelenioNext Broadcast Management System V.1.0 50 Installation and Troubleshooting Guide
Appendix A – Creation of a System Definition File
To generate a system definition file, create a copy of the sample system definition file (/usr/share/ibs/scripts/setup/xmls/<SAMPLE_TYPE>), preferably using a different name, and edit it to represent your system.
There are files for the following possible system types, in addition to the generic file:
Dell R720; <SAMPLE_TYPE> = system-drac.sample
Generic, without remote management cards; <SAMPLE_TYPE> = system-bmc.sample
This system definition file should be stored on a SelenioNext server and run as a “qoduser” user.
Sample Dell R720 Definition File #! /usr/bin/ruby /usr/bin/rvt .template /usr/share/ibs/scripts/setup/xmls/system.rxml
.output system-drac.xml
cluster-id Dell cluster-mcast-ip 255.255.255.255
cluster-port 16642 data-netmask 255.255.0.0
mgmt-netmask 255.255.0.0 mgmt-gw 172.16.0.1
ntp-ip 172.16.200.113 idp-ip 255.255.255.255
dynamic-routing yes
data0-if eth0 data1-if eth3
mgmt-if eth2
h: hostname cluster-ip data1-stub-ip mgmt-stub-ip stonith
h bms 10.200.1.56 10.200.2.56 172.16.203.56 ipmi:172.36.200.224:root:qodadmin
h next-1 10.200.1.124 10.200.2.124 172.16.200.124 ipmi:172.16.200.224:root:qodadmin
h next-2 10.200.1.125 10.200.2.125 172.16.200.125 ipmi:172.16.200.225:root:qodadmin
r: group-id mgmt-ip data0-ip data1-ip hosts
r dell-1 172.16.200.24 10.200.1.24 10.200.2.24 next-1, next-2
Maintenance Procedures
SelenioNext 51 Installation and Troubleshooting Guide
The above sample for Dell represents a 1+1 cluster, and therefore there is an entry for the BMS server.
Sample Generic System Definition File (2 data interfaces) #! /usr/bin/ruby /usr/sbin/rvt .template /usr/share/ibs/scripts/setup/xmls/system.rxml
.output system-bmc.xml
cluster-id ibs1 cluster-mcast-ip 255.255.255.255
cluster-port 16642
data-netmask 255.255.0.0
mgmt-netmask 255.255.0.0 mgmt-gw 172.16.0.1
ntp-ip 192.168.11.253 idp-ip 255.255.255.255
dynamic-routing no
data0-if eth0 data1-if eth3
mgmt-if eth2
h: hostname cluster-ip data1-stub-ip mgmt-stub-ip stonith
h ibp1 10.190.1.41 10.190.1.141 172.16.190.41 ipmi:172.16.190.11:root:qodadmin h ibp2 10.190.1.42 10.190.1.142 172.16.190.42 ipmi:172.16.190.12:root:qodadmin
h ibp3 10.190.1.43 10.190.1.143 172.16.190.43 ipmi:172.16.190.13:root:qodadmin h ibp-standby 10.190.1.44 10.190.1.144 172.16.190.44 ipmi:172.16.190.14:root:qodadmin
r: group-id mgmt-ip data0-ip data1-ip hosts
r ibp1 172.16.190.31 10.190.1.31 10.190.1.131 ibp1,ibp-standby r ibp2 172.16.190.32 10.190.1.32 10.190.1.132 ibp2,ibp-standby
r ibp3 172.16.190.33 10.190.1.33 10.190.1.133 ibp3,ibp-standby
Maintenance Procedures
SelenioNext Broadcast Management System V.1.0 52 Installation and Troubleshooting Guide
Notes:
The system file needs to be executable: use “chmod a+rx <file>”
cluster-mcast-ip – Do not change this parameter; it must be set to 255.255.255.255.
cluster-port is the port being used by the cluster , default is 16642. The value of cluster-port is in the following range : 1024 < cluster-port<=65535
dynamic-routing is dynamic binding of multicast source to physical interface. When set to yes, the source can be joined from a different interface.
In order to alter interfaces used for primary and secondary data and mgmt-if, uncomment the lines such as # data0-if by removing the hash mark, and edit the interface names.
The h: table represents the physical hosts and their static details (hostname and IPs). These are the primary IPs.
Host Resources
Host resources include:
Hostname
Cluster-ip is the IP address that is assigned to data0-ip, and is used to transfer heartbeat messages between the servers
Data1-stub-ip is assigned to data1-if, and is used to keep the data1 interface active to save failover time
Mgmt-stub-ip is assigned to mgmt-if, and is used to keep the mgmt interface active to save failover time
Stonith are the parameters for Stonith (shutting down a potentially malfunctioning server): method (IPMI/Telnet/Shoot), IP, username, password (if required). In some cases, several methods may be supplied.
The r: table represents the resources managed by the cluster, which are moved between the hosts during a failover event. The last column in the table represents which hosts each resource can be assigned to, the first value representing the default. These are the “Virtual IPs”.