MSR Command Cell Facility

WashingtonWASHINGTON UNIVERSITY IN ST LOUIS

[email protected]

MSR Command Cell Facility

Fred Kuhns

Applied Research Laboratory

2WashingtonWASHINGTON UNIVERSITY IN ST LOUIS

Fred Kuhns - 1/9/01

Highlights• Overview

• High level description - Application Layer

• MSR Command Interface Overview

• Cell format and field definitions

• Example


Fred Kuhns - 1/9/01

Definitions• Session: Open connection between the CP

and a specific SPC. Intended to represent open connections and command state

• Transaction: Represent a complete command. A transaction terminates with either an EOF is received by the CP or and error occurs.

• EOF: End of File is returned to CP with the last bit of command data is returned or in response to a Cancel message (or an error occurs)


Fred Kuhns - 1/9/01

Overview - Cmd Interface on CP• Synchronous Request/Response protocol

• Timeout can be specified as well as the number of retries - Per session option– Essentially provides a reliable service– Issue: if no reply, cmd/reply msg lost in port,

channel or CP. Retries may be a bad thing.

• Address - MSR Port and Command – <MSR_Port, MSR_Command>

• Message destination - Callback function within the Port’s kernel (implements command)


Fred Kuhns - 1/9/01

Command Interface on CP• Types of messages:

– New Command, Get Next set of reply data Command, Cancel Command

– Error Reply, EOF Reply, Continued Reply

• Message Identifiers - Only requires a sequence number initialized to 0 for each New Command:– One sending entity on CP, – One outstanding command for each port,– Ports send exactly one reply msg per command msg,– Command must fit within one cell,– Replies may span multiple cells.


Fred Kuhns - 1/9/01

Command Interface on Port• Callback function registered with MSR kernel

and called under 3 cases:– New Command

• Flags = Command; Sequence = 0; Length = valid bytes in buffer; Buffer = application data

– Next Command • Flags = Command | Next; Sequence = previous+1; Length

= valid bytes in buffer; Buffer = application data

– Cancel Command • Flags = Command | Cancel; Sequence = previous+1;

Length = 0; Buffer contains no valid data


Fred Kuhns - 1/9/01

Command Interface on Port• Callback function must:

– Read from/Write to supplied buffer– Set length = Bytes written to buffer (in/out param)– Indicate if an error occurred (return -1)– Whether more data exists (return 0 => EOF, return > 0

=> Not EOF, return < 0 => ERROR | EOF)

• Framework: – generates reply message using same Command value

and Sequence number.– sets flags indicating status (EOF, Error etc)


Fred Kuhns - 1/9/01

Failure Modes• Library support for lost messages:

– if (timeout > 0, Replies > 0), then CP API library will re-send with RETRY flag set.

– if (timeout > 0, Replies = 0 or all replies failed), then API library returns error to application

– If (timeout = 0 - No Timeout), then send operation blocks indefinitely.

• Lost Command message - – if (timeout > 0 and retries > 0), CP resends command;

same sequence number but RETRY flag set. Command buffer and flags passed to callback fn.


Fred Kuhns - 1/9/01

Failure Modes• Lost Reply message,

– if no retries, Any issues?– if retries then CP resends

• New Command - Port knows this is a duplicate command (RETRY flag). Application responsible for handling retries. If an issue can use unique message ids. Extreme case use a history (last reply message).

• Next Command - Port receives Command w/Sequence > 0, w/RETRY flag. Passed to application which chooses the correct course of action. The intent is to ensure there are no holes in the reply data received by the CP.

• Cancel message - same as Next command.


Fred Kuhns - 1/9/01

Possible Enhancements• Support asynchronous messaging:

– Multiple outstanding commands per port– Asynchronous I/O on CP– Speed up boot process and dynamic configuration– Facilitates implementing port monitoring (ping or

heartbeat) for fault detection and recovery. – two methods for reporting results:

• upcall - function registered by application is called when results arrive

• poll - application periodically polls library for results.

• Support Broadcast and/or Multicast


Fred Kuhns - 1/9/01

MSR Command Layer• Simple messaging facility optimized for MSR.

• Command message (CP sends):– Sent by CP to a specific MSR port (unicast)– Must fit within one AAL0 cell.– Message header, includes:

• protocol version

• Command

• Sequence number

• flags

– Application data follows header– Library implements Request/Reply protocol.


Fred Kuhns - 1/9/01

MSR Command Layer• Reply Message (Port sends):

– Port must send reply message in response to a Command message.

– Reply message Header:• version and sequence number: same as command msg.

• Includes application data and flags indicating if command was successful and if more data exists (EOF).

– Application registers command specific callback function at port.

– Callback function must conform to specified interface.


Fred Kuhns - 1/9/01

MSR Command Overvew

• Command Protocol description– Control Processor sends command messages to a

specific port and expects to receive a reply message indicating either Success or Failure. This is termed a Command Cycle.

– There is the notion of a Command Transaction which may include one or more command cycles. A command transaction is terminated when the target (port) responds with a reply msg containing an EOF


Fred Kuhns - 1/9/01

MSR Command Overvew• Command Protocol description, continued

– CP processing of Reply msg, depends on EOF flag:• If EOF is set then no further reply data is available and

the command transaction is closed.

• If EOF is not set then there is remaining data and the command transaction is still open.

– If remaining data (Not EOF), then CP must follow with a either a Next or Cancel command message.

• Sequence number indicates the “chunk” of data to be returned.

• Command indicates the message’s destination

• sequence number = previous + 1


Fred Kuhns - 1/9/01

APIC Cell FormatM

SR C

omm

and

Mes

sage

81624 0cidldccpoutx xxpin

clgfc vpi vci pti

x x x x x x x x x x x x x

ver length command/status sequence number flags

• Cell payload contains the MSR Command• Command header is 4 Bytes, leaving 44 Bytes

for sub-commands and data.


Fred Kuhns - 1/9/01

ATM/APIC Header

• pin (Ports-In) - Port cell arrived– Tx not used (set to 000b)– Rx: port cell arrived on (is the below correct?)

• 001 Port 0, 010 Port 1, 100 Port 2, etc.

• pout (Ports-Out) - Set of output ports.– Tx: Command library sets:

• 001 Fiber/Link, 010 Ribbon/Switch, 011 Both

• 101 Loopback MV0, 110 Loopback MV1

– Rx: Set by VCXT, see pin above.

cidldccpoutx xxpin

clgfc vpi vci pti



Fred Kuhns - 1/9/01

ATM/APIC Cell Format

• cc (Control Cell Indicator) - Not used, set to 0b

• ld (Low Delay) - Not used, set to 0b. – Should we use low delay?

• cid (Connection Identifier) - set to vci value.

• gfc (Generic Flow Control) - set to 0000b.

cidldccpoutx xxpin

clgfc vpi vci pti



Fred Kuhns - 1/9/01

ATM/APIC Cell Format

cidldccpoutx xxpin

clgfc vpi vci pti


• vpi (Virtual Path Identifier) - Set to 0x0.

• vci (Virtual Circuit Identifier) - Equal to cid. – See presentation on MSR configurations for a

complete list of VCI assignments.

• pti (Payload Type) - Set to 000b (data cell)

• cl (Cell Loss Priority) - Set to 0b (High Priority)


Fred Kuhns - 1/9/01

MSR Command Header

• Version (2 bits) - Protocol version. Allows for at most 4 versions. Current version set to 0.– field width was a trade off with the length field.

• Length (6 bits) - Number of valid data bytes.– 0 <= Length <= 44, so 6 bits sufficient.– This field is indirectly set by the application or

command implementation. The CP library and kernel interfaces allow for applications to pass a buffer pointer and indicate the number of valid data bytes.



Fred Kuhns - 1/9/01

MSR Command Header

• Command/Status (8 Bits) CP inserts command value, SPC/port inserts status information.– Valid Commands are listed in $SYS/msr/msr_ctl.h,

also see $MSR/utils/command/*.{c,h}– Library API on CP accepts Command as argument.

implementation in kernel - array of function pointers, uses Command as index

– Reply msg Status indicating success or an error code (Upcall, ATM, Cmd Invalid, Cmd Not Implemented, or Other Cmd Error).



Fred Kuhns - 1/9/01

Current Commands• Current set of commands:

– set_debug/get_debug - set debug flags/masks. – port_init - set local port number.– policy - Manage MSR policy object. Currently

sets the Policy control flags (Gate 0 enable)– dq - Preshanth?– fipl -Ed?– rp_pcu - extensive set of sub-commands.– rp_inst - send message to plugin instance– rp_class - send message to plugin base class


Fred Kuhns - 1/9/01

MSR DEBUG Command• Dynamic setting of debug mask - affects

what messages are sent to syslogd

• set_debug or get_debug command– Valid debug categories/modules

• apic, ipfwd, iprx, iptx, mem, dq, stats, ctl, conf, kern, natm, pcu, plugin, gate

– Valid debug levels - 0 - 255• predefined: verbose, warning, error, critical

• Interface in MSR kernelMSR_DEBUG((MSR_DEBUG_<category>|MSR_DEBUG_LEVEL_<level>, “format”, args));


Fred Kuhns - 1/9/01

MSR RP_PCU Command• rp_pcu Command sub-commands

– addfltr/remfltr - add/remove pkt filter at gate x– flist - port prints current gate x filter list– bind/unbind - bind/unbind instance to fltr/gate combo– create - create plugin instance– free - remove plugin instance– clist - port prints current plugin class list– ilist - port prints current instance list– load - load plugin - not implemented – null - no-op. Can be used as a ping operation


Fred Kuhns - 1/9/01

MSR Command Header

• Sequence Number (8 Bits) - Is of primary use by the applications. – When command message first sent, sequence = 0.– If the reply does not include an EOF flag, then CP

increments sequence by one for each subsequent command message.

– When EOF is received the Command Transaction is complete and the sequence number is reset to 0.



Fred Kuhns - 1/9/01

MSR Command Header

• Flags (8 bits) - Bit field, valid flags are:– Invalid - flag = 0, should not occur– CMD - cell contains a valid command from CP– REPLY - cell contains reply from Port– ERROR - Reply only, error processing on Port – EOF - No reply data remains, end of cmd transaction– NEXT - get next set of reply data– CANCEL - cancel current cmd transaction– RETRY - set if cp resend a command after it was lost



Fred Kuhns - 1/9/01

CP Library API• Library API for application on CP,

– int sendcmd(int sid, int cmd, char *data, int flags, int *dlen)

• sid = session id, • cmd - Command to execute on port• data = buffer pointer, • flags =

– RETRY (reply timeout),

– CANCEL (cancel current command),

– Next (get next set of reply data)


Fred Kuhns - 1/9/01

MSR Kernel API• MSR kernel interface $SYS/msr/msr_ctl.{h,c}• Callback function signature:

– msr_ctl_<cmd> (void *buf, u_int8_t flags, u_int8_t seq, u_int8_t *dlen)

– buf = command buffer w/application data,

– flags = • CMD,

• NEXT,

• RETRY or

• CANCEL,

– seq = sequence number indicating reply data set, and

– dlen is input/output parameter, data length in bytes.


Fred Kuhns - 1/9/01

Kernel State Diagram

Command

Closed

Next Retry

Idle

Retry

Com

man

d

Proto

Error

CommandEO

F

Proto Error

Cance

l


Fred Kuhns - 1/9/01

CP Library State Diagram

Wait (for reply)

Closed

Next Retry

Idle

Proto

col E

rror Command

EOF

Protocol Error

Open Session

Result of a timeout


Fred Kuhns - 1/9/01

Example Sending Cmd to Port

CP

Next/PrevHop

Next/PrevHop

Next/PrevHop

Next/PrevHop

Next/PrevHop

Next/PrevHop

Next/PrevHop

wugsP0

P1

P2

P3

P4

P5

P6

P7

192.168.200.X

192.168.201.X

192.168.202.X

192.168.203.X

192.168.204.X

192.168.205.X

192.168.206.X

192.168.207.XSPC/FPX

SPC/FPX

SPC/FPX

SPC/FPX

SPC/FPX

SPC/FPX

SPC/FPX

SPC/FPX

DQ

DQ

DQ

DQ DQ

DQ

DQ

DQ

192.168.203.2

192.168.202.2

sendcmd();create plugin instance:

port id = 0,PluginID = 200

cmddata

cell hdr

msr_ctl

reply();plugin instance created:

Status,Instance ID

Report command completion status

to application.

Lookup sub-commandperform function call

then report results

Documents

MSR Command Cell Facility