Debugging Advance

Debugging Simulation Models —AdvancedNetwork R&D

Session 1503

CONFIDENTIAL ─ RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without the prior written consent of OPNET Technologies, Inc.

© 2008 OPNET Technologies, Inc.

CONFIDENTIAL ─ RESTRICTED ACCESS: This information may not be disclosed, copied, or transmitted in any format without the prior written consent of OPNET Technologies, Inc. © 2008 OPNET Technologies, Inc. 2

1503 Debugging Simulation Models ─ Advanced

1502 - Debugging Simulation Models ReviewWhy use a C/C++ debuggerUnderstanding process model codeHow to debug a program abortScientific approach to debuggingUsing a C/C++ debugger with ODBAdvanced C/C++ debugger techniquesUsing execution traces to validate changesMemory leak tracking

Agenda



1502 Review – Dissecting an Error Message

|----------------------------------------------------------------|| This node does not have any valid IP interfaces || Assign a valid address to at least one interface || T (0), EV (1593), MOD (top.DCI Network.FR_Cloud.ip), || PROC (ip_dispatch_intf_table_create (total_interfaces)) ||----------------------------------------------------------------|

Event Number Hierarchical Name

Function NameSimulation Time



1502 Review – Function Call Stack

* Function call stack: (builds down)------------------------------------------------------------

Call BlockCount Line# Function

------------------------------------------------------------0) 1 152 0xd84a6f08 [name not available]1) 1 2211 0x00004c00 [name not available]2) 1 1358 0x0000c400 [name not available]3) 1 291 m3_main4) 1 1106 sim_main5) 1 2706 sim_ev_loop6) 1565 531 sim_obj_qps_intrpt7) 69 15 ip_dispatch [wait -> cmn_rte_tbl :

SELF_NOTIFICATION / ip_dispatch_init_phase_2 ()]8) 9 1791 ip_dispatch_init_phase_2 ()9) 9 7364 ip_dispatch_intf_table_create (total_interfaces)

10) 1 649 op_sim_error (gravity, line1, line2)------------------------------------------------------------

Kernel



Block Line Numbers

21 static void22 ip_encap_pk_destroy (Packet* pkptr)23 {24 Ici* intf_iciptr;25 IpT_Rte_Ind_Ici_Fields* intf_ici_struct_ptr;2627 /** Destroys the IP datagram received from lower **/28 /** layer and the associated ip_rte_ind_v4 ICI **/29 FIN (ip_encap_pk_destory (pkptr));3031 /* Get the ICI associated with the packet. */32 intf_iciptr = op_pk_ici_get (pkptr);3334 /* Destroy the ICI and its fields */35 if (intf_iciptr != OPC_NIL)36 {37 op_ici_attr_get (intf_iciptr,38 "rte_info_fields", &intf_ici_struct_ptr);39 ip_rte_ind_ici_fdstruct_destroy (intf_ici_struct_ptr);40 op_ici_destroy (intf_iciptr);41 }4243 /* Destroy the packet. */44 op_pk_destroy (pkptr);4546 FOUT;47 }



FIN, FOUT, and FRET Macros

How does OPNET know what to put in the function call stack?• Process model states – automatic• Need to add a macro for complete functions

Function BlockExternal C/C++ codePipeline Stages

FIN - Macro used to label a function• C: Place after local variables• C++: First statement in the function

When using FIN macro, replace return keyword with•FOUT replaces return;•FRET(value); replaces return(value);

Using FIN without FOUT / FRET in C results inStandard Function Stack Imbalance error messageNo overhead when models compiled with comp_trace_info preference set to FALSE (Optimized Kernel compile)FIN required in Function Block functions if using state variables



Why Use a C/C++ Debugger?

Benefits• View values and types of C/C++ variables• View execution at a more fine-grained level• Data-based breakpoints

Caveats• Requires familiarity with C/C++ • Requires familiarity with OPNET generated code

Proper C/C++ compilation flags for debugging are set by default automatically when working with development kernel

NOTE: “C/C++ Debugger” also referred to as “Symbolic Debugger” or “Source-Level Debugger”



Example Debuggers

Separate GUI

Integrated w/OPNET GUI

Integrated w/OPNET GUI

UseName Platform Tutorial

CDB Windows This session

GDB Linux OPNETWORK 2006

Visual Studio Windows OPNETWORK 2005



Compiling a process model generates a C/C++ file• <model_name>.pr.c• <model_name>.pr.cpp

Process Model Code Structure

Header Block

State Variable Structure

State Variable Macros

Function Block

void <model name> ()

Temporary Variables

Finite State Machine

_op_<model name>_diag ()



Process Model Code Structure (cont.)

Header block inserted at top of file• Header file #includes• Global definitions (variables, typedefs, etc.)




State variables structure declaration block follows the header blockState variables packed into a single structure that is pointed to by the op_sv_ptr variable•op_sv_ptr is implicitly declared and obtained from the simulation kernel

by the FIN statements• Any function that use state variables MUST start with a FIN statement




Function block inserted above primary function• State variable definitions available




Temporary variable block → local variables




Primary function has same name as model

Executives and transitions are implemented as macros




Diagnostic, termination blocks → helper functions• Temporary variable block → local variables• Other helper functions generated by OPNET



Compiler Warnings

Results of kept in:• <home>/op_admin/tmp/cc_err_<compiled process id>

Compiler warnings can identify:• Uninitialized variables• Non-prototyped functions•‘=‘ in test, or ‘==‘ not in test• Statements with no effect•printf type mismatches…

Preference Show Compilation Output to TRUE• Always show compiler output instead of just when there are errors

char * char_ptr;strcpy (char_ptr, "hello world");

warning C4700: local variable 'char_ptr‘ used without having been initialized



Debugging a Program Abort

Run simulation, see an abort message

Make sure code is compiled with debugging symbols (automatic in development kernel).CDB/GDB• Run simulation with development kernel and ODB enabled• In simulation execution window, select

Windows: Simulation > Attach Windows Debugger (CDB)Linux: Simulation > Attach GDB

• Hit Continue button in console window, source code will appear on abort



Debugging a Program Abort (cont.)

Visual Studio• In Configure/Run DES dialog, select Execution -> Troubleshooting and set

Prevent OPNET from handling and logging its exceptions to Yes• Run simulation • The following dialog box will come up. Select Debug to launch Visual Studio

and attach it to the crashed simulation process• Dialog will not appear if CDB or another source debugger is already attached



Lab 1: Using a Debugger to Investigate a Simulation Abort

Description:Run a simulation, and notice that the simulation crashes. Use a C/C++ debugger to diagnose the problem and find a solution.

Recap• C/C++ debuggers can help tracking down program aborts by bringing you

to the point of the abort




Agenda



The Approach: Scientific Method

Create controlled environment • Reproducibility is critical

Establish the knowns (define the error)Collect circumstantial evidenceDevelop hypothesesDesign experiments• Understand what each experiment will answer• Does it or does it not confirm hypothesis?

Run experiments (one at a time)Iterate as necessary



Difficult to Debug Problems

Problem cannot be reproduced consistently• Memory corruption• Uninitialized variables

Problem does not appear when using debugger• Side-effects of debugger (printing, tracing, displaying variables) alters

environmentProblem only occurs late in a large simulation



Debugging Techniques

Goal: Make the problem occur at an earlier time and systematicallyApproach: Destabilize the systemTechniques• Vary simulation seed• Change object order (cut/paste)• Change attribute or parameter values• Change environment (OS, 32/64 bit, compiler version)•-mem_optimize FALSE (Disable OPNET memory management

optimizations)•-mem_shred TRUE (Set contents of freed memory to 0xDF)• Use third-party memory corruption detection tools such as purify, valgrind etc.

Make sure to use -mem_optimize FALSESearch OPNET FAQs for particular tool integration instructions



Using a C/C++ Debugger With ODB

In order to stop a debugger at a point in a simulation that is not an abort:

1. Run the simulation with the OPNET Debugger enabled2. Wait for the simulation to load and stop before the first event3. Attach the C/C++ debugger to the simulation4. Set breakpoints in ODB or the C/C++ debugger5. (CDB/GDB) Click Continue in Simulation Execution dialog6. (Visual Studio only) Continue C/C++ debugger, then Continue in ODB

Using ODB and the C/C++ debugger together1. ODB stops on event boundaries2. C/C++ debugger stops on lines of code3. Set breakpoints in both, simulation will run to whatever breakpoint is next4. (Visual Studio only) Continuing in C/C++ debugger may bring you back to a

paused state in ODB



Conditional Breakpoints

Problem:• You want to set a breakpoint at a line of code that is executed quite often.• You only want to break on line of code under a certain situation

Solution: Conditional breakpoints. • The breakpoint is triggered only if a defined C/C++ expression evaluates to

TRUECDB• Command: bp• Example:

GDB• Command: “break” to set a breakpoint, followed by “cond” to assign a

condition • Example:

break my_process.pr.c:383 cond <breakpoint ID> (op_sv_ptr->my_process_id == 49)



Conditional Breakpoints (cont.)

Visual Studio• Define a new breakpoint (Ctrl-B, or click on New in the Breakpoint window)• Click on Condition…• Type in conditional expression (op_sv_ptr->my_process_id == 49)



Watchpoints

Problem: You want simulation to stop when a variable value changes unexpectedly; this is a common source of bugsSolution: Use watchpoints. They stop execution of a simulation when a chosen variable value changesHow to set a watchpoint on a state variableCDB (refer to Lab 2)GDB (refer to OPNETWORK 2006 Lab 3)GDB> print &op_sv_ptr->my_tpal_objid<address>GDB> watch *((int *) <address>)



Watchpoints (cont.)

Visual Studio• Define a new breakpoint (Ctrl-B, or click on New in the Breakpoint window)• Select the Data tab• Enter expression to watch in the Variable: field• Example: *(int *) 0x038472fc



Getting Debugger Help

CDB

GDB: Type ‘help’ at command promptVisual Studio: Help menu



Lab 2: Using a C Debugger and ODB ─Advanced

This lab shows advanced C/C++ debugger techniques:• Conditional breakpoints• Watchpoints

Difficult error: bug and error message are separated by many events

Long lab with tricky parts

The most advanced lab at this conference



Lab 2: Recap

Error reported when models was runDeveloped two hypotheses: • Variable initialized improperly (demonstrated to be false)• Variable changed unexpectedly elsewhere (demonstrated to be true)

Used conditional breakpoint to stop at a particular processUsed watchpoint facility to pinpoint when bug occurredConditional breakpoints and watchpoints can turn good debuggers into expert debuggers



Agenda




Using Execution Trace to Validate Changes

Model changes / optimizations typically not supposed to change the resultsFinding the exact point where two simulations that supposed to produce identical results begin to diverge can be difficultExecution Trace captures basic simulation kernel actions such as packet creation and interrupt schedulingSimulation execution can be compared against a previously captured Execution Trace. Simulation will stop at the first encountered difference in basic simulation actions



Execution Trace Discrepancy Report Example



Execution Trace Configuration

Run simulation with “-etrace_dump <FILE_PATH>” on the command line to capture the traceRun simulation with “-etrace_diff <FILE_PATH>” on the command line to compare execution against the captured traceAdditional options to trim the range of events that are captured / compared:•“-etrace_start_time <SIM_TIME>” will start capturing / comparing

actions once simulation time goes past the specified number•“-etrace_end_time <SIM_TIME>” will stop capturing / comparing

actions once simulation time goes past the specified number



Lab 3: Using Execution Trace to Validate Changes

The task is to validate a model change with little knowledge about the model or the nature of the changeUse simulation Execution Trace to find an event where simulations that for some reason produce different results divergeUse ODB trace to find a piece of source code that is responsible for the differenceFix the code



Agenda




Problem: Memory Leak

Missing deallocation of memory no longer referenced by the simulationApproach• Identify what type of memory is leaking• Identify sources of allocations (modules and function call stacks)

Use the Memory Usage and Memory Stats tabs in Simulation dialog• Statistics are updated live in the Simulation Progress Window as simulation

runsUse advanced simulation memory tracking features • Tracks simulation objects by module

source and function call stack source• Memory Usage tab shows per-category

memory growth



Memory Sources: Track Active Memory with Allocation Source

View source of memory allocation by process state / functionSort by Bytes to identify largest memory usersDefault grouping by process state machine sourceInvert function call stack to group by offending KP



Memory Sources:Block by Block

New blocks since last update appear in RED



Memory Sources:Configuration

Or, enter specific categories:

Track all sources:



Lab 4: Using Advanced Memory Features in ODB

Use the memory tracking features from within the Simulation Execution dialog to track down a memory leakUse Memory Stats and Memory Usage tabs on simulation dialog to track memory problems within the Project EditorUse advanced memory tracking to view the creator of simulation objects that might be leaking



Documentation References

ODB in the Simulation Execution chapter of the External Interfaces Manual

Process Model Domain Definition, Modeling Concepts ManualYour C/C++ debugger manual



Additional Optional Labs

From OPNETWORK 2005: Debugging on Visual C++ (6.0), Visual Studio .NET 2003, UNIX debugging using dbx (Solaris)From OPNETWORK 2006: Debugging on Linux



Supported Compiler and Debuggers

gdb 6.3 and newergcc 3.4 and newer

Visual C++ 6, Visual Studio .NET 2003, 2005 and 2008Microsoft Debugging Tools for Windows version 6.6.7.5 or newer

Intel compiler ICL 8.0 or newer OK on Windows and Linux (not currently supported, but works)



Consider Linux

Reach set of free and reliable software development toolsGDB commands are more intuitive than CDB commandsCheckpoint / Restart simulation feature is only available on LinuxVirtual Machine is an efficient way of using the same PC for both bureaucracy under Windows and serious simulation debugging underLinuxSame OPNET license covers all platforms• OPNET license server can serve the same OPNET licenses to both Windows

and Linux clients



Some Useful Debugging FAQs on Support Site

841: When using a C/C++ debugger, how do I set a breakpoint for a particular state’s enter/exit execs195/1219/1686: What are the proper system environment settings for Visual C++ 6.0/.NET 2003/.NET 20051667: How can I use gdb, dbx or other UNIX source level debuggers to debug simulations (apart from gdb integration)517: How can I use Purify with OPNET to debug a memory corruption, memory leak, or crash? For identical simulation runs, why does my simulation crash in different places, maybe even "inside" the simulation kernel? 554: How can I solve the problem that my simulation uses a large amount of memory? What is some general advice for tracking down memory leaks or queues that grow without bound?



OPNET Solutions for Debugging Simulation Models ─ Advanced



Take-Away Points

Any OPNET simulation is easily debugged with a source code debuggerTake advantage of the ODB and source debugger integrationAnalyze errors to start the debugging processUse the Scientific Method to debugAdvanced debugger features (conditional breakpoints, watchpoints) turn a difficult debugging problem into an easy debugging problemExecution tracing is a powerful model change validation toolMemory tracking is a useful tool for detecting memory leaksNext steps: • Build simulation with debug information• Next bug that comes up will put you in the debugger

Documents

Debugging Advance