LTE_Symptom_Saving_and_Tools Guideline_v2.0.11.pdf

LTE Symptom Saving & Tools Guideline

All rights reserved – Internal Use Only

1 / 30

LTE Symptom Saving &

Tools Guideline (RL40) for NSN TS Team

Author Pietro Sapienza ([email protected])

MBB Customer Support

Status APPROVED

Version 2.0.11

Location IMS

mailto:[email protected]

https://sharenet-ims.inside.nokiasiemensnetworks.com/Open/490537344



2 / 30

Revision History

Version Date Author Comments

2.0 (Draft) 29.01.2013 Pietro Sapienza Upgrade to RL40

2.0.1 (Draft) 01.03.2013 Pietro Sapienza Martin Ruff’s feedback

2.0.2 (Draft) 07.03.2013 Pietro Sapienza Serhiy Tertyshnikov’s feedback

2.0.3 (Draft) 08.03.2013 Pietro Sapienza Basant Dhakal’s feedback

2.0.4 (Draft) 12.03.2013 Pietro Sapienza Table 2 added

2.0.5 (Draft) 19.03.2013 Pietro Sapienza BTSLog Remote Connection

2.0.6 (Draft) 20.03.2013 Pietro Sapienza BTSLog 2.2.1.x update

2.0.7 20.03.2013 Pietro Sapienza Doc title changed

2.0.8 25.03.2013 Pietro Sapienza Review

2.0.9 26.03.2013 Pietro Sapienza BTSLog IMS Link for FSMr2

2.0.10 27.03.2013 Pietro Sapienza Review

2.0.11 28.03.2013 Pietro Sapienza MAC TTI Trace sizes



3 / 30

Table of Contents

Revision History .............................................................................................................................................. 2

Table of Contents ........................................................................................................................................... 3

Index of Figures .............................................................................................................................................. 5

Index of Tables ............................................................................................................................................... 6

Glossary .......................................................................................................................................................... 7

1. Introduction .............................................................................................................................................. 8

1.1. Scope ................................................................................................................................................ 8 1.2. Keywords .......................................................................................................................................... 8 1.3. Summary .......................................................................................................................................... 8

2. Description ................................................................................................................................................ 9

2.1. Hardware Failure ............................................................................................................................ 10 2.2. System Failure or Problem ............................................................................................................. 10 2.3. Improvement proposal ................................................................................................................... 10

3. Symptom Collection ................................................................................................................................ 11

3.1. Operation & Maintenance Problems ............................................................................................. 11 3.1.1. eNB Snapshot ............................................................................................................................ 12

3.1.1.1. Snapshot preconditions .................................................................................................... 12 3.1.1.2. How to fetch eNB Snapshot ............................................................................................. 12 3.1.1.3. Alarm History .................................................................................................................... 13 3.1.1.4. Check SW Version ............................................................................................................. 13 3.1.1.5. Fault Diagnosis Log via BTSSM ......................................................................................... 14

3.1.2. R&D Parameters ....................................................................................................................... 15 3.1.2.1. Connect FCM .................................................................................................................... 15 3.1.2.2. Check active file system ................................................................................................... 15 3.1.2.3. Edit swconfig .................................................................................................................... 15 3.1.2.4. Change R&D flags without reboot .................................................................................... 16

3.2. Call Processing Problems ................................................................................................................ 16 3.2.1. Emil ......................................................................................................................................... 17

3.2.1.1. Installation ........................................................................................................................ 17 3.2.1.2. Configuration .................................................................................................................... 19 3.2.1.3. Activate Snapshot Filter ................................................................................................... 20 3.2.1.4. Start/Stop Tracing ............................................................................................................ 21

3.2.2. BTSLog ....................................................................................................................................... 21 3.2.2.1. Installation ........................................................................................................................ 22 3.2.2.2. Configure and Logging ...................................................................................................... 22 3.2.2.3. Remote Syslog .................................................................................................................. 23 3.2.2.4. Troubleshooting ............................................................................................................... 25

3.2.3. MAC TTI traces .......................................................................................................................... 25 3.2.3.1. MAC TTI Uploader ............................................................................................................. 25



4 / 30

3.2.4. Wireshark Traces ...................................................................................................................... 27 3.2.5. Port mirroring for FSMr2 .......................................................................................................... 27

3.2.5.1. Setup port mirroring ........................................................................................................ 27 3.2.5.2. Start/Stop EIF2 monitoring............................................................................................... 28

3.2.6. LMP mirroring on FSMr3 .......................................................................................................... 28 3.2.6.1. Activate port monitoring .................................................................................................. 28 3.2.6.2. Stop monitoring................................................................................................................ 29

3.2.7. IP traffic on FTM using tcpdump ............................................................................................... 29

4. Conclusions ............................................................................................................................................. 30

Disclaimer ......................................................................................................................................... 31

Analytic Index ............................................................................................................................................... 32

References .................................................................................................................................................... 33

Annex 1 ......................................................................................................................................................... 34

BTSLog config files............................................................................................................................... 34 Available services ................................................................................................................................ 34



5 / 30

Index of Figures

Figure 1. BTSSM screenshot #1. ................................................................................................................... 12 Figure 2. BTSSM screenshot #2. ................................................................................................................... 12 Figure 3. BTSSM screenshot #3. ................................................................................................................... 13 Figure 4. CurrentBD.xml file screenshot. ...................................................................................................... 13 Figure 5. BTS Site Manager screenshot. ....................................................................................................... 14 Figure 6. BTS Log Setting dialog in BTSSM. ................................................................................................... 14 Figure 7. BTS Log Setting dialog (Log Available). .......................................................................................... 14 Figure 8. R&D active flags. ............................................................................................................................ 16 Figure 9. Emil installation pop-up message. ................................................................................................. 17 Figure 10. Open your SACK folder. ............................................................................................................... 18 Figure 11. Email SACK pop-up message. ...................................................................................................... 18 Figure 12. Select a SACK and a profile. ......................................................................................................... 18 Figure 13. Emil Options pop-up window. ..................................................................................................... 19 Figure 14. BTS connections tab. .................................................................................................................... 19 Figure 15. CP+UP_Light profile. .................................................................................................................... 20 Figure 16. RL40 FSMr3 2 Pipe (default Profile). ............................................................................................ 20 Figure 17. Snapshot trigger configuration. ................................................................................................... 21 Figure 18. Monitoring Mode: Local. ............................................................................................................. 22 Figure 19. Log Configuration. ....................................................................................................................... 22 Figure 20. Syslogs. ........................................................................................................................................ 23 Figure 21. Logging configuration. ................................................................................................................. 23 Figure 22. R&D Parameters. ......................................................................................................................... 24 Figure 23. R&D param setup. ....................................................................................................................... 24 Figure 24. Editing of Index 65 and 66. .......................................................................................................... 25 Figure 25. MACTTIUploader v6.0 screenshot. .............................................................................................. 26 Figure 26. MACTTIUploader.ini screenshot. ................................................................................................. 27



6 / 30

Index of Tables

Table 1. Case Types and Priorities in Resolve. .............................................................................................. 10 Table 2. Local vs. remote trace collection. ................................................................................................... 11 Table 3. Overview on Symptoms for OAM related problems (Legenda in footnote ). ................................. 11 Table 4. Overview on Symptoms for CP related problems (Legenda in footnote ). ..................................... 17 Table 5. MAC TTI Trace Sizes for RL40. ......................................................................................................... 27



7 / 30

Glossary

BTSSM BTS Site Manager

FD Fault Diagnosis

HW Hardware

LMP Local Maintenance Port

NE Network Element

NSN Nokia Siemens Networks

NWS Network Systems

R&D Research and Development

TS Technical Support

For completeness refer to the following document for a wider glossary: BTS_Glossary.doc

https://sharenet-ims.inside.nokiasiemensnetworks.com/livelink/livelink?func=ll&objId=396829964&objAction=Open&vernum=39&nexturl=%2Flivelink%2Flivelink%3Ffunc%3Dsrch%2ESearchCache%26cacheId%3D295145030



8 / 30

1. Introduction

1.1. Scope This document contains generic information about NSN products. These can be instructions that explain problem situations in the field, instructions on how to prevent or how to recover from problem situations. It is indented to be a useful reference document for Care Engineers and TCP inside Nokia Siemens Networks.

1.2. Keywords Problem report, Resolve, LTE outage, System restart, Unit restart, symptom collection, Emil, BTS Site Manager, BTS Snapshot, MAC TTI trace, Port mirroring, R&D Parameters.

1.3. Summary This document contains basic instructions for making a problem report in such a way that all the necessary information is included in it. Filling in the problem report carefully and attaching required information to it makes the problem investigation process faster.



9 / 30

2. Description

This document contains instructions for collecting basic data from the network element (eNB) for fault analyzing purposes. This data shall always be collected as soon as some abnormal situation occurs in the system, and NSN will be informed of it. An abnormal situation may be, for example, a decreased traffic handling capacity, or a spontaneous restart of a unit. In the worst case the abnormal situation may lead to a system outage. The data to be collected is essential for the further analysis of the problem and for trying to find out the fault root causes. An immediate data collection is important, since information about the problem (e.g. alarm history) may get overwritten during the subsequent runtime period. The data to be collected, as well as the commands to be used for collecting them, have been described below. The data shall be stored in descriptively named log files, which shall then be zipped into to a file and delivered to NSN as an attachment of the problem report or outage report. After data collection, fill in the problem report (in Resolve) by taking into account the following issues:

1. Title of the problem report should be a very short description of the fault situation. 2. Description of the problem in the problem report should be an exact and a clear description of the fault situation. Note that only one fault shall be reported for each problem report and at least the following information should be provided in addition to the actual problem description:

i. Situation in the beginning, e.g. the first symptoms of the failure. ii. Operations made, which possibly caused the failure.

iii. Situation after the failure. iv. fault frequency (e.g. permanent, occasional, one occurrence) v. Recovery actions made.

vi. Name and version of the possibly installed new software modules.

3. Make sure that necessary attachments are included to the problem report to avoid unnecessary and time consuming information requests. 4. In a multivendor environment, collect detailed information of other products to the Description field of the problem report. 5. Write down the SW Release used on the NE. 6. Write down the SW build of the SW Release eNB (see § 3.1.1.4).



10 / 30

7. For filling in the severity of the problem, see the definition of the different severities in Table 1, NSN’s Problem Classification:

Priority in Resolve

Corresponding Pronto Severity

Priority examples

Critical Emergency A loss of service comparable to the total loss of effective functional capability of an entire switching or transport system. A reduction in capacity or traffic handling capability such that expected loads cannot be handled. Any loss of safety or emergency capability.

Major Critical Reduction in any capacity/traffic measurement function. Any loss of functional visibility and/or diagnostic capability. Short outages equivalent to system or subsystem outages (accumulated duration > 2 minutes in any 24-hour period, or repeated during longer periods). Repeated degradation of DS1 or higher rate spans or connections. Prevention of access for routine administrative activity. Degradation of access for maintenance or recovery operations. Degradation of the system’s ability to provide any required critical or major trouble notification. Any significant increase in product related customer trouble reports. Billing error rates that exceed specifications Corruption of system or billing databases.

Medium Major Less significant failure of the deliverables such as processor restarts with no traffic impact. Some adverse impact on the network or a relevant subsystem of the network that affects call processing, traffic handling or subscriber service in some manner. Serious inefficiency of O&M functionality. Documentation roblems that cause a serious inefficiency in the O&M process.

Minor Minor Non-traffic affecting situations.

Table 1. Case Types and Priorities in Resolve.

2.1. Hardware Failure When you suspect that a failure or problem is caused by the hardware, but you cannot locate the faulty unit, you can report the fault using the problem reporting practice. If you find a faulty unit, fill in the hardware failure report, attach it to the faulty unit, and send them to your local Technical Support. Set the case type of the problem report to value Hardware.

2.2. System Failure or Problem The most common use of the problem report is reporting a system defect or problems related to the software or data configuration. Set the case type of the failure report to value Documentation.

2.3. Improvement proposal When there is no actual fault, but you want to suggest some improvement to the way the network element functions, you can do this using problem reporting. New feature proposals should be directed directly to Product Marketing. Set the case type of the problem report to value General.



11 / 30

3. Symptom Collection

In this chapter we will introduce some practical instructions for symptom collection. Logs as well as trace are required by submitting a new Resolve case. Concerning O&M issues, we will explain step-by-step how to collect logs via BTS Site Manager (see §3.1). On the other side, dealing with Call Processing issues, log collection via Emil (see §3.2.1) and BTSLog (see §3.2.2) will be introduced. In Table 2, we sum up which traces can be capture locally and/or remotely.

Connection

Log/Trace Local Remote

O&M

eNB snapshot LMP port (p. 12) M-Plane IP (p. 12)

zcollectlogs See §6.1.21 -

pcap trace See §6.1.31 -

OMS config export See §6.1.51 -

CP

Emil LMP port (p. 20) M-Plane IP (p. 20)

BTSLog LMP port (p. 21) M-Plane IP for FSMr3 (p. 23)

MAC TTI LMP port (p. 25) M-Plane IP (p. 27)

Wireshark EIF3 port (see §3.2.5) -

tcpdump LMP Port (see §3.2.7) -

Table 2. Local vs. remote trace collection.

3.1. Operation & Maintenance Problems In Table 3, logs and traces related to O&M problems are listed.

Scenario O&M Related issues

NE Log/Trace Live test Lab test

eNB eNB snapshot ● ●

iOMS zcollectlogs --full (debug) ● ●

pcap trace (Wireshark) ● ●

NetAct

OMS config export ● ●

Export plans ● ●

Table 3. Overview on Symptoms for OAM related problems (Legenda in footnote 2).

1 LTE iOMS Problem Report Instructions” (TS-LTE_iOMS-SW-0003-I4) is available in NOLS.

https://online.portal.nokiasiemensnetworks.com/pic/com/nsn/extranet/pic/controller/productinfoview/pivdocdetails.do?callBackAction=pivdocumentation&docId=urn:nsn.com:sharenet:443219124&productId=urn:nsn.com:mxpdm:134-004636



12 / 30

For iOMS, “LTE iOMS Problem Report Instructions” (TS-LTE_iOMS-SW-0003-I4) is available in NOLS.

3.1.1. eNB Snapshot The eNB Snapshot contains the following information:

Active alarms and alarm history for eNB (see §3.1.1.3) Installed SW versions eNB configuration (commissioning parameters) Internal log files for different eNB components (including MAC TTI traces) PM counter data for this eNB (starting from the last eNB reset, but max for 24 hours)

3.1.1.1. Snapshot preconditions BTS Site Manager must be properly installed and connected to the eNB either locally on the Local Maintenance Port (LMP) or remotely using the M-Plane IP address of the eNB.

3.1.1.2. How to fetch eNB Snapshot The eNB snapshot can be saved by means of the BTS Site Manager either by clicking on the camera icon in the toolbar (see Figure 1) or by the command File > Save > Snapshot (see Figure 2).

Figure 1. BTSSM screenshot #1.


In the Save Snapshot dialog some context information should be provided (e.g. which test has been performed before the snapshot was saved) as shown below. Also it is a good idea to add the time to the snapshot name if several snapshots will be saved for one site at the same day (see Figure 3). NOTE. About Snapshot coverage, requirements are described in Testability SFS. These requirements are not very strict and there is nearly no difference between full coverage and most important data (e.g. MAC TTI traces are included into both of them). If you collect the current data (see Detail level in Figure 3), it will only include the eNB configuration and some BTSSM logs. Actually it makes sense to collect full coverage snapshots in all cases. The current problem is that it takes long time for Snapshot collection. This is addressed by: “CN5491 – compression algorithm change” and “CN5193 – file aggregation”. Unfortunately even with all the efforts and pressure, these features are planned for RL60 (possibly RL50), whereas Snapshot collection will take approx. 5 minutes.

2 ● = mandatory, ● = strongly recommended, ● = recommended.

https://online.portal.nokiasiemensnetworks.com/pic/com/nsn/extranet/pic/controller/productinfoview/pivdocdetails.do?callBackAction=pivdocumentation&docId=urn:nsn.com:sharenet:443219124&productId=urn:nsn.com:mxpdm:134-004636



13 / 30


3.1.1.3. Alarm History The alarm history shows all alarm events, occurred on eNB. You can open alarm history, locating correspondent txt files inside the Snapshot zip file

3. Alarm history files, called BTS*_AlarmHistory.txt

and BTS*_RawAlarmHistory.txt, are stored into /LTEBTS/BTSLogFiles/ folder.

3.1.1.4. Check SW Version The active SW version, running on the eNB, can be read out within the Snapshot. Check the file, called CurrentBD.xml in /LTEBTS/SWVersions/ (see Figure 4).

Figure 4. CurrentBD.xml file screenshot.

Besides, it can be read out also from BTS Site Manager, under the BTS Hardware tab (see Figure 5).

3 E.g. Snapshot_T1079_20110913.zip.



14 / 30

Figure 5. BTS Site Manager screenshot.

3.1.1.5. Fault Diagnosis Log via BTSSM This procedure describes how to configure the automatic creation of the eNB Fault Diagnosis Log

4 using

the BTS Site Manager. The eNB Fault Diagnosis Log is a file located in /tmp directory on the FCMD and named FD_Trap.log.gz before upload to the BTSSM

5. Such file is automatically generated after that a configurable BTS fault

occurs. It is possible to configure up to 5 BTS faults. FD Log embodies SYSLOG messages from FCMD and shows what was going on before a certain BTS fault occurrence. For configuring the BTS faults that should trigger the FD Log creation, open Tools > BTS Log Setting dialog and select Fault Diagnosis tab (see Figure 6).

Figure 6. BTS Log Setting dialog in BTSSM.

Figure 7. BTS Log Setting dialog (Log Available).

Insert up to 5 BTS fault numbers as Trigger number and submit them by Send. If you choose Permanent as Validity, the BTS fault numbers will be stored on the FCMD within the file /rom/swconfig.txt as shown here:

368 = 10

369 = 6150

370 = -1

371 = -1

372 = -1

4 Twiki reference here.

5 This file log is called BTS_1011_FD_Trap.log within the BTS snapshot.

https://twiki.inside.nokiasiemensnetworks.com/bin/view/LTETechSup/HowToAutomaticallyCreateFaultDiagnosisLog



15 / 30

Once one of the configured BTS faults has occurred, the file FD_Trap.log.gz is created on the FCMD /tmp directory. The user is notified about, if the BTS Log Setting dialog is opened, as shown in Figure 7. You can save the Snapshot directly from this dialog. The snapshot will contain the FD log (named BTS_1011_FD_Trap.log) within the BTSLogFiles.zip archive. Moreover BTS Log Setting dialog offers the possibility to set which Log Data Level the Snapshot should embrace: minimum, normal or full

6.

3.1.2. R&D Parameters Based on problem severity, log filtering is possible by means of R&D params. In fact, the latter turns out to be extremely useful to get more traces from eNB. If enabled, R&D params will trigger a more extensive syslog output. All R&D params to control Debug Logging behavior are boolean: thus allowed values are 0 (disabled) or 1 (enabled)

7.

As precondition, the following instructions require a PC connected to FCM LMP port (or a connection via FTP). To enable R&D parameters, swconfig.txt file on FCM has to be edited on purpose.

3.1.2.1. Connect FCM Connect FCM via telnet

8 with the following socket: IP 192.168.255.1 and Port 23.

Once telnet session has been established, enter into FCM by using username and password:

FCMD login: toor4nsn

Password:

whereas password field has to be filled up with the well-known FCM password.

3.1.2.2. Check active file system Verify which file system is currently running on eNB, as follows:

root@FCMD:/ >ls -l /ffs/

drwxr-xr-x 10 root root 1976 Jan 2 1970 fs3/

drwxr-xr-x 10 root root 2088 Aug 4 21:33 fs4/

lrwxrwxrwx 1 root root 8 Jan 1 1970 run -> /ffs/fs4/

root@FCMD:/ >

The file system /ffs/fs4 is therefore active (from the example above).

3.1.2.3. Edit swconfig Check if swconfig.txt is present, as follows:

root@FCMD:/ >ls -l /ffs/fs4/swconfig.txt

-rw-r--r-- 1 root root 271 Aug 4 15:00 /ffs/fs4/swconfig.txt

Then edit it with Vi9 by entering the following command:

root@FCMD:/ >vi /ffs/fs4/swconfig.txt

The swconfig.txt file must be structured as in the example below:

1 # TEnbcRadDebugLogEnabledCommon

2 0x110003 0

3

4 # TEnbcRadDebugLogEnabledEnbc

5 0x110002 1

6

6 From minimum vs. full level comparison, the only difference is the presence of MAC TTI traces into the full-levelled Snapshot.

7 All other values will be ignored.

8 Use Putty or TeraTerm.

9 Or any other Linux-compatible text editor. Vi Tutorial: here.

parameter description

tag (hex)

flag active

flag inactive

http://acms.ucsd.edu/info/vi_tutorial.html



16 / 30

whereas R&D flags are sorted out according to their correspondent system component belonging (CELLC, ENBC, RROM, TUPC, UEC). R&D flags are provided on purpose according to which traces are required. After editing, save and close the swconfig.txt and reboot the system module, by entering:

root@FCMD:/ > reboot

After reboot, the so edited R&D parameters will be active and the correspondent traces will be recorded into logs (e.g. BTSSM Snapshots, see §3.1.1).

3.1.2.4. Change R&D flags without reboot Connect to FCM via telnet, as described in §3.1.2.1, then connect to Aashell as follows:

telnet 192.168.255.1 15007

To read out active R&D flags, type:

rad –r 0 0

You will see the active flag list, as depicted in Figure 8. In order to edit/set an R&D flags, execute the command:

rad -mw <domain> <value>

Such change will be active until next eNB reset. Using –mpw as option (instead of –mw), every change will be permanently saved.

Figure 8. R&D active flags.

3.2. Call Processing Problems The following symptoms are related to CP problems (see Table 4):

Scenario Call Establishment Failures /

Call Drops / HO E2E Throughput/Latency

NE Log/Trace Live test Lab test Live Test Lab Test

eNB eNB snapshot ● ● ● ●

eNB-EPC

eNB-eNB

S1-C, X2-C (Wireshark) ● ● ● ●

S1-U, X2-U (Wireshark) ● ●

Uu Interface UE trace (basic) ● ● ● ●

UE trace (advanced) ● ● ● ●

Terminal Equipment

Drive test report ● ●

Wireshark trace ● ●



17 / 30

Application logs10

● ●

Application Server

Wireshark trace ● ●

Application logs10

● ●

Table 4. Overview on Symptoms for CP related problems (Legenda in footnote 11).

3.2.1. Emil Emil turns to be a reliable tool to handle big file sized traces, to trace BTS message remotely, to decode LTE-Uu CP, S1 CP and X2 CP interface protocols, to fetch data from eNB immediately after a failure event. Snapshots are many times needed in case of UP problems or FSP/eNB outages. Moreover Emil is able to monitor and analyze multiple LTE BTS, depending on LTE BTS load. In customer networks preferred way of taking Emil traces and Snapshots is to use the Emil Customer version

12. The latter will show only Emil tracing window with currently embedded SACK version (e.g.

LN2.0_ENB_1009_290_14). Moreover it replaces all IMSI and IMEI digits with “0” for legislation reasons. WARNING! Since Emil shows internal messaging between SW system components, do not share such information with the customer.

3.2.1.1. Installation Inside NSN Intranet

13, installation package can be downloaded from:

https://isource.access.nokiasiemensnetworks.com/projects/emil/ Here the direct link to Emil v2.0.9.9 (file size: ~36MB) on iSource. Install it on your PC. After the installation a notification is shown to specify a directory which contains the required SACKs:

Figure 9. Emil installation pop-up message.

Retrieve appropriate SACKs from link to Sacks. Specify the directory where the SACK packages are stored locally on the PC:

10

jperf, ping, etc. 11

● = mandatory, ● = strongly recommended, ● = recommended. 12

LTE1091 BTS Diagnostics Toolkit. 13

Access the above-mentioned link with your NSN-INTRA account.

https://isource.access.nokiasiemensnetworks.com/projects/emil/

https://isource.access.nokiasiemensnetworks.com/frs/download.php/13434/Emil.msi

https://twiki.inside.nokiasiemensnetworks.com/bin/view/LTETechSup/Internal-Tools#Sacks_required_for_using_Emil_ID



18 / 30

Figure 10. Open your SACK folder.

"A sack and a profile is needed" notification pops up, click “OK”:

Figure 11. Email SACK pop-up message.

Select the appropriate Sack, specify the profile LTE and select Load:

Figure 12. Select a SACK and a profile.

The selection can be changed afterwards by selecting a new SACK directory via Tools > Options > Sack Path and then by Environment > Change sack and profile. For further installation instructions, please have a look to this video tutorial #1. Start Emil and configure your log paths from Tools > Options (see Figure 13).

https://sharenet-ims.inside.nokiasiemensnetworks.com/livelink/livelink?func=ll&objId=429372648&objAction=Open&viewType=1&nexturl=%2Flivelink%2Flivelink%3Ffunc%3Dll%26objId%3D429377248%26objAction%3DBrowse



19 / 30

Figure 13. Emil Options pop-up window.

3.2.1.2. Configuration Open Emil and import BTS configuration by clicking on File > Connect to LTE BTSs. Now parameters in the window, depicted in Figure 14, have to be entered.

Figure 14. BTS connections tab.



20 / 30

In Target, enter the save path where you intend to save Emil logs. On first installation, you get always an Example entry as default BTS name (with IP 10.10.10.10). Accordingly to your BTS network configuration, you can add further BTS by right-clicking on the list and choosing Add new bts. Enter BTS name and BTS IP on the right column, while TCP port, Monitor via board and CPU can be left as the default ones. In case of thousands of BTS in your network, import them via File > Import BTS list

14.

On the second tab Monitoring profiles, choose which profile you are going to use to trace messages from BTS (see Figure 15). Each profile embodies one or more intervals of message identifiers, that will be used as message filter in tracing. To see the calls, use the default profile, called CP+UP_Light.

Figure 15. CP+UP_Light profile.

In case of FSMr3 (FSMF) HW, select the proper profile15

as depicted in Figure 16.

Figure 16. RL40 FSMr3 2 Pipe (default Profile).

In case of remote connection, check if PC is connected to same sub network as eNBs and that IPSec and firewall setting allow tracing from M-Plane IP Address. In case of local LMP connection, instead, set PC IP address as 192.168.255.126 and enable LMP use from Site Manager security settings. Trace ports need to be opened in IPSec and Firewall settings. Note: In the default profile, all FSP are expected to be configured. If they are not, you will get the “partially connected” indication. However Emil can trace as well.

3.2.1.3. Activate Snapshot Filter Needless to say the automatic snapshot fetching turns out to be useful in many faulty situations. Before start tracing, thus, select trigger conditions from Filter > Set snapshot filter for TCP. The most common events for call drops are selected in Figure 17, whereas blue-boxed parameters refer to the frequency of alarm occurrence and they can be set up according the chosen condition, the orange-boxed ones are the maximum number of occurrence per BTS and, finally, the green-boxed area addresses to BTS Snapshot coverage, if Fetch BTS snapshot is checked, the drop down menu allows to choose how deep the Snapshot collection has to be.

14

If you do not have already a BTS list in .csv format, please watch the final part of video tutorial #1. 15

E.g. RL40 FSMr3 2 Pipe (default Profile).




21 / 30

Figure 17. Snapshot trigger configuration.

Once trigger conditions have been set up, activate filter from Activate filter under Filter top bar menu.

3.2.1.4. Start/Stop Tracing Connect all BTS from BTS connection tab under File > Connect all LTE BTSs (see Figure 14 above) by clicking on Connect all button (top right Connection box). Besides, you can connect single BTS by right-clicking on it and choosing Connect from the context menu; moreover from the same context menu, you can fetch some BTS information (e.g. Fetch technical logs, Fetch SW version, Send ping, etc.). WARNING! Check that BTS Site Manager is not tracing same eNBs at the same time. BTSSM will prevent Emil Snapshots as well, since they use same O&M IP address. After running, when you stop (e.g. by Disconnect all), you get the Call KPI Data in a pop-up message window, which sum up tracing results in one shot. Click OK. Now the traces have been saved and you can see start/stop times from Emil main window

16. Now:

Check the fetched logs from your log directory (.emil files). Zip the complete sub directory where your BTS traces are Create new ticket for new discovered faults or attach more detailed and timely logs for existing

pending tickets. Inform about new logs and agree of regular log update interval

If a problem situation has been observed, it is best practice to:

Run traces continuously or during defined intervals, until problems are captured/solved; Continue tracing in current eNB cluster or move or expand tracing to a new cluster for analyzing

possible problem situations; Provide traces and logs to NSN Technical Support.

3.2.2. BTSLog BTSLog is the main tool for collection of internal traces and for logging BTS in real-time. The communication is based on the SICAP over TCP protocol stack for message trace and UDP for log entries. The SICAP protocol allows BTSLog to be directly connected to the system internal communication SW bus. BTSLog is therefore capable to communicate with SW components (applications, services) inside the BTS. Some literature is available on twiki and inside btslog_tutorial.pdf appended to installation files.

WARNING! BTSLog shows internal eNB messaging, do not share such information with the customer.

16

For further installation instructions, please have a look to this video tutorial #2.

https://twiki.inside.nokiasiemensnetworks.com/bin/view/BtsTestabilityWeb/BTSLog




22 / 30

3.2.2.1. Installation To install BTSLog, perform the following steps:

- Download the installation file from IMS (BTSlog2-2.1.x.msi)17

- Follow BTSLog installation steps (described in details here)

3.2.2.2. Configure and Logging To capture syslogs from FSMx in LOCAL mode, follow the steps below:

- Open BTSLog and select Local as Monitoring Mode, as depicted in Figure 18

Figure 18. Monitoring Mode: Local.

- Under Actions > Options, choose Log Configuration - Set up file size and output folder (see Figure 19) and click OK

Figure 19. Log Configuration.

- From the main window, click Start inside UDP box - Click 1 button inside Real Time Views to open the window depicted in Figure 20

17

For previous BTSLog versions, refer to this twiki link.

https://sharenet-ims.inside.nokiasiemensnetworks.com/livelink/livelink/overview/D490532133

http://ultima.inside.nokiasiemensnetworks.com/ee_release_notes/btslog_2_1_0/BTSlog_step_by_step.html

https://twiki.inside.nokiasiemensnetworks.com/bin/view/LTETechSup/HowToRemoteBTSLog



23 / 30

Figure 20. Syslogs.

3.2.2.3. Remote Syslog What described within this paragraph concerns only FSMr3 (FSMF). Concerning FSMr2, refer to the existing documentation on IMS. As precondition, stop any tracing and logging activity.

1. Select your output directory from Log Configuration tab under Actions > Options 2. Enable SSH Service and R&D Port Service via cgi scripts in your web browser

18 (with Nemuadmin

credentials):

https://<M-Plane IP>/protected/enableSsh.cgi

https://<M-Plane IP>/protected/enableRndPorts.cgi

Note: Both scripts have to be relaunched in case of eNB reset. For further details about cgi scripts, see Available services in Annex 1.

3. Enable SysLog over TCP from Configuration tab under Actions > Options menu (see Figure 21) and click OK

Figure 21. Logging configuration.

18

For LN4.0, Microsoft Internet Explorer cannot be used on Win XP PC, because AES during SSL handshake is not supported; use Mozilla Firefox.

https://sharenet-ims.inside.nokiasiemensnetworks.com/Open/431229146



24 / 30

4. Select View > R&D Parameters view from main window 5. Select WAM_10 inside Target node box (see Figure 22) 6. Select WN50_CCS inside SW Products (see Figure 22) 7. Select Ccs inside Domains (see Figure 22)

Figure 22. R&D Parameters.

8. Click Setup in Target node box 9. Set M-Plane IP address (e.g. 10.46.163.113) and port (i.e. 15004) 10. Set NMAP address (Board 0x10, CPU 0x11, Task 0x060c) as depicted in Figure 23

Figure 23. R&D param setup.

11. Click Get Current 12. From the table below, change

19 the Index 65 and 66 (as depicted in Figure 24):

ERadCcs_AaSysLogOutputMode (RAD_CCS 65) = 1

ERadCcs_AaSysLogUdpAddress (RAD_CCS 66) = <your PC IP address>

13. Click Set New in order to send R&D Parameters to your BTS20

14. Click Start inside UDP box (main window) to begin log collection 15. Click Stop to end log collection

19

Simply double-click the row to edit it. 20

You can click again Get Current in order to double-check your new settings.



25 / 30

Figure 24. Editing of Index 65 and 66.

3.2.2.4. Troubleshooting Some possible problems, concerning BTSLog logging, can be troubleshot as follow: - Bad UDP port configuration. Check UDP port configuration in UDP Ports under Actions > Option. - Firewall blocking. In Win7, try to disable TrendMicro Firewall for your IP address (e.g. 192.168.255.126).

3.2.3. MAC TTI traces Purpose of LTE MAC TTI trace is to get information about LTE MAC SW behavior running on target HW (e.g. the behaviour of the packet scheduling software). TTI

21 trace provides means to get data on TTI basis

(in every 1ms.) which would not be possible with written ASCII logs without significant performance losses. TTI Traces for both UL and DL are commonly appended to BTSSM Snapshot (as seen in §3.1.1). However, due to eNB ring buffer

22 limited capacity, MAC TTI Uploader turns out to be useful to capture

such TTI traces in a wider time window.

3.2.3.1. MAC TTI Uploader For uploading MAC TTI trace files (in binary format) from the eNB, you can use the following tool:

InstallMACTTIUploaderV6.0.zip

Follow the installation procedure and launch MACTTIUploader after installation.

In this first section part, we consider a local scenario. Thus your PC is assumed directly connected to FSMx via LMP port. Afterwards, the remote case will be discussed.

21

Transmission Time Interval. 22

Ring buffer (8-10 MB) holds data approx. 15-60 sec (UL) until it is over written.

https://twiki.inside.nokiasiemensnetworks.com/pub/LTETechSup/Internal-Tools/InstallMACTTIUploaderV6.0.zip



26 / 30

If eNB IP address23

is not pingable, you will get an error pop-up message: therefore troubleshoot your eNB connection. Otherwise the tool will be ready as in Figure 25.

Figure 25. MACTTIUploader v6.0 screenshot.

This version supports both FSMr2 and FSMr3 (default) eNBs, whereas different core IDs are used to store MAC TTI traces, respectively. For tool point of view, only if you use FSME (aka FSMr2):

- Right-click on window area and select Change eNB FSM Type item from context menu - Then, select Display INI file from same menu - Edit CPU core IDs in macttiuploader.ini file, as follows:

…

# MAC TTI downlink data faraday#2 (SICFTP): No_eNB_FSP1_FD2=1241

No_eNB_FSP2_FD2=1341


# MAC TTI uplink data faraday#4 (SICFTP):




…

…

# FSME MAC TTI downlink data core IDs (SICFTP):




# FSME MAC TTI uplink data core IDs (SICFTP):




…

To start tracing:

- Choose proper trace destination path (1st

window box in Figure 25) - Select which cell you want to trace (2

nd window box in Figure 25)

- Choose upload mode (3rd

window box in Figure 25)

WARNING! On FSMF eNBs with current RL40 SW loads, parallel upload of MACTTI trace files works one-by-one. Therefore do not use parallel upload mode (aka DL+UL par.), but select one mode among the following ones: DL+UL seq., DL only or UL only.

23

I.e. 192.168.255.1.



27 / 30

- Click Upload MACTTI button (inside 4th

window box in Figure 25)

Optionally, you can choose the Automatic Upload and set it accordingly.

In remote case scenario, some settings have to be applied to MACTTIUploader.ini located into installation folder

24:

- Open MACTTIUploader.ini with your text editor - Edit it as depicted in Figure 26

Figure 26. MACTTIUploader.ini screenshot.

- Save and close the file - (Re)start MACTTIUploader.exe.

In RL40, memory sizes for DL and UL MAC TTI traces differ according to Table 5.

MAC TTI Trace Sizes [MB]

Downlink Uplink

FSMr2 10 10

FSMr3 10 8

Table 5. MAC TTI Trace Sizes for RL40.

3.2.4. Wireshark Traces Wireshark is currently our main LTE S1 and X2 analyzer tool. All information about it are clearly reported in twiki: Wireshark for LTE and How to configure.

3.2.5. Port mirroring for FSMr2 The following procedure is described in twiki

25 and is going to be summed up below.

3.2.5.1. Setup port mirroring Current procedure is based on the assumption that EIF2 is used and EIF3 is free for monitoring:

1. Connect an Ethernet cable from PC to FSMr2 (use LMP port).

24

E.g. C:\Program Files (x86)\MACTTIUploader\. 25

https://twiki.inside.nokiasiemensnetworks.com/bin/view/LTETechSup/TracingIPTrafficOnFTMUsingPortMirroring

https://bts.inside.nokiasiemensnetworks.com/twiki/bin/view/Testlab/WL_WiresharkLte

https://bts.inside.nokiasiemensnetworks.com/twiki/bin/view/Testlab/WL_WiresharkConfig

https://twiki.inside.nokiasiemensnetworks.com/bin/view/LTETechSup/TracingIPTrafficOnFTMUsingPortMirroring



28 / 30

2. Enable telnet connection to FTM vie Web Browser26

.

3. Open SSH session to FTM on 192.168.255.129 as toor4nsn user.

4. Activate EIF3 on FTM:

test_ml2Drv -u -n p=eif3

5. Check if EIF3 is up:

ml2lstat

6. Add port mirroring for EIF2 and send it to EIF3 (ingress direction):

ml2monitor -i add=eif2 -I set=eif3

7. Add port mirroring for EIF2 and send it to EIF3 (egress direction):

ml2monitor -e add=eif2 -E set=eif3

3.2.5.2. Start/Stop EIF2 monitoring Start monitoring as follows:

1. Connect an Ethernet cable from PC to FTM EIF3 port27

. 2. Start Wireshark tracing.

Stop monitoring as follows:

1. Disable EIF2 on FTM:

test_ml2Drv -u -n p=eif3

2. Remove port mirroring on EIF2:

ml2monitor –c

3.2.6. LMP mirroring on FSMr3 For the following procedure, eNB SW has to be LN4.0_ENB_1202_579_00 or later

28.

IMPORTANT! Egress traffic is not mirrored (PR 88943ESPE01). Feature LTE 1432 shifted from RL40 P8

branch to RL40 MP1 branch. Refer to §3.2.7 as workaround.

The target port for mirroring is LMP. The mirroring is applicable to all ingress and egress IP traffic for the

system's Transport interfaces, except user plane traffic. The measurement point is located after IPsec

decryption (ingress) resp. before IPsec encryption (egress), i.e. the mirrored traffic is not IPsec-encrypted.

3.2.6.1. Activate port monitoring 1. Connect an Ethernet cable from PC to LMP port of FSMF. 2. Enable telnet connection to FTM vie Web Browser

29.

3. Open SSH session to FTM on 192.168.255.129 as toor4nsn user. 4. Enable traffic mirroring, run the following command in the shell:

/opt/trs/bin/secfg_test iptest_enable_traffic_mirror_feature <destination MAC> <timeout>

26

http://192.168.255.129/protected/EnableTelnet.cgi 27

Use a direct cable connection between EIF3 and PC and don't route the mirrored traffic through the same switch as

the original traffic; this would most likely cause problems within the switch. 28

Related feature is LTE1432 - Integrated traffic mirroring for FSMr3. 29

http://192.168.255.129/protected/EnableTelnet.cgi





29 / 30

E.g. to mirror traffic to destination MAC address30

00:40:43:00:00:01 for 120 seconds:

/opt/trs/bin/secfg_test iptest_enable_traffic_mirror_feature 0x004043000001 120

On connected PC use Wireshark for tracing (see §3.2.4 for Wireshark details).

3.2.6.2. Stop monitoring To disable traffic mirroring, run the following command in the shell:

/opt/trs/bin/secfg_test iptest_disable_traffic_mirror_feature <destination MAC>

3.2.7. IP traffic on FTM using tcpdump For the following procedure is also described in twiki

31.

Note: C-Plane traffic (e.g. S1AP messages) cannot be captured by TCPDump on FTM.

1. Download the tcpdump-binary to your local machine:

tcpdump_for_FTM_Westport.zip

2. Connect your PC to FTM LMP and configure your IP settings accordingly. 3. Transfer

32 the tcpdump-binary via FTP to /var/log/ on FTM.

4. Activate telnet on FTM33

. 5. Open telnet connection to FTM (on 192.168.255.129 as Nemuadmin user). 6. Change permissions of tcpdump-binary:

chmod 755 /var/log/tcpdump

Now you can start dumping traffic on the different interfaces. E.g. dump all traffic from eth1:

/var/log/tcpdump -i eth1

E.g. Dump all traffic from eth1 interfaces into a file34

, which can be analyzed by Wireshark:

/var/log/tcpdump -i eth1 -w /tmp/capture.cap -s 0

30

Destination MAC address is the one of the Network Adapter used on PC connected to LMP port. 31

https://twiki.inside.nokiasiemensnetworks.com/bin/view/LTETechSup/TracingIPTrafficOnFTMUsingTcpdump 32

You can use standard Windows FTP client (with Nemuadmin credentials) on IP 192.168.255.129. 33

See footnote 29. 34

The file can be fetched always via ftp.

https://twiki.inside.nokiasiemensnetworks.com/bin/view/LTETechSup/TracingIPTrafficOnFTMUsingTcpdump



30 / 30

4. Conclusions

The so far discussed topics are intended to help readers in daily eNB troubleshooting. Although many extensive additional considerations can be appended to the present guide, completeness and exhaustiveness cannot be the target of those few pages. Thence criticism and feedback are welcome, in order to make these pages even more useful and clearer for future users.



31 / 30

Disclaimer The information in this document is subject to change without notice and describes only the product defined in the introduction of this documentation. This documentation is intended for the use of Nokia Siemens Networks internal employees only for the purposes of the agreement under which the document is submitted, and no part of it may be used, reproduced, modified or transmitted in any form or means without the prior written permission of Nokia Siemens Networks. The documentation has been prepared to be used by professional and properly trained personnel. Nokia Siemens Networks welcomes employee comments as part of the process of continuous development and improvement of the documentation. Nokia Siemens Networks has made all reasonable efforts to ensure that the instructions contained in the document are adequate and free of material errors and omissions. Nokia Siemens Networks will, if deemed necessary by Nokia Siemens Networks, explain issues which may not be covered by the document. Nokia Siemens Networks will correct errors in this documentation as soon as possible. IN NO EVENT WILL NOKIA SIEMENS NETWORKS BE LIABLE FOR ERRORS IN THIS DOCUMENTATION OR FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO SPECIAL, DIRECT, INDIRECT, INCIDENTAL OR CONSEQUENTIAL OR ANY LOSSES, SUCH AS BUT NOT LIMITED TO LOSS OF PROFIT, REVENUE, BUSINESS INTERRUPTION, BUSINESS OPPORTUNITY OR DATA, THAT MAY ARISE FROM THE USE OF THIS DOCUMENT OR THE INFORMATION IN IT. This documentation and the product it describes are considered protected by copyrights and other intellectual property rights according to the applicable laws. The wave logo is a trademark of Nokia Siemens Networks Oy. Nokia is a registered trademark of Nokia Corporation. Siemens is a registered trademark of Siemens AG. Other product names mentioned in this document may be trademarks of their respective owners, and they are mentioned for identification purposes only. Copyright © Nokia Siemens Networks 2013. All rights reserved.



32 / 30

Analytic Index

B

BTS Site Manager .............................................. 11 BTSLog .............................................................. 21

E

EIF2 ................................................................... 27 EIF3 ................................................................... 27 Emil ................................................................... 17 Emil logs ............................................................ 20

F

Fault Diagnosis .................................................. 14 FCM via telnet .................................................. 15 FD Log ............................. See Fault Diagnosis Log

L

LMP ............................................................. 12, 27 Log Data Level .................................................. 15

M

MAC TTI traces...................................... 12, 15, 25

MAC TTI Uploader............................................ 25 mirroring ............................................... 27, 28, 29

P

Priorities in Resolve ......................................... 10

R

Resolve............................................................... 9

S

SACKs ............................................................... 17 Snapshot .......................................................... 12 Snapshot coverage .......................................... 12 symptom collection ......................................... 11 syslogs .............................................................. 22

W

Wireshark ........................................................ 27



33 / 30

References

BTSLog: https://confluence.inside.nokiasiemensnetworks.com/display/EE/EE+BTSlog IDA2: https://confluence.inside.nokiasiemensnetworks.com/display/EE/EE+IDA2 Automatic Creation of eNB Fault Diagnosis Log: https://twiki.inside.nokiasiemensnetworks.com/bin/view/LTETechSup/HowToAutomaticallyCreateFaultDiagnosisLog Tools and Methods for Troubleshooting & Debugging in LTE Network: IMS

https://confluence.inside.nokiasiemensnetworks.com/display/EE/EE+BTSlog

https://confluence.inside.nokiasiemensnetworks.com/display/EE/EE+IDA2



https://sharenet-ims.inside.nokiasiemensnetworks.com/livelink/livelink/overview/D426842465



34 / 30

Annex 1

BTSLog config files

nodes.ini

Available services For FSMr2 EnableTelnet.cgi and DisableTelnet.cgi enable/disable telnet/SSH on FTM external interface EnableRemoteFcmSSH.cgi and DisableRemoteFcmSSH.cgi enable/disable SSH PORT 15008 or standard PORT 22 on FTM external interface EnableRemoteFcmTestPorts.cgi and DisableRemoteFcmTestPorts.cgi enable/disable PORT 15003 LTE browser PORT 15004 BTS Log For FSMr3 enableSsh.cgi and disableSsh.cgi enable/disable standard SSH PORT 22 on FTM external interface enableRndPorts.cgi and disableRndPorts.cgi enable/disable PORT 15003 LTE browser PORT 15004 BTS Log

Documents

LTE_Symptom_Saving_and_Tools Guideline_v2.0.11.pdf