IPC Time Measurements Reporting

Design

During the development of OpenSync software, frequently there was a need to know the real-time duration of particular procedures, stages, or phases that run on network nodes. It was particularly required to know the period of time for stages which are part of bigger procedure such as node startup, onboarding, network interface configuration, etc.

Feature purpose: providing the time duration data in real-time. The data can be collected from every OpenSync node and can be used for continuous monitoring of:

• Network node startup time

• Configuration change time

• Detailed time dependency analysis

Q&A may use such time measurements for fast, reliable and detailed detection of configured network delays. R&D may use the measurements to track down the cause of network and device configuration slowdown cases not only in development units but also on client premises running with nodes.

System View

In this design, the network node processes are allowed to create EVENTS in user space, which are then transmitted via network to the remote location named subscriber in the MQTT terminology.

This design by nature is very similar to the syslog Linux feature. Therefore, the design follows a simplified architecture of syslog.

Client/Server architecture

Each process is allowed to create an event client object (EvClient). EvClients uses IPC to communicate with single instance of events server (EvServer).

EvClient shall provide methods to create and send EVENTS to EvServer.

EvServer shall accept incoming connections from EvClients and when connection is established, EvServer shall be ready to receive EVENTS from EvClients.

This design creates architecture with multiple EvClients connected to single EvServer, with multiple writers (of EVENTS) and single destination server. This kind of relation creates requirement for synchronization of EVENT writes from multiple clients to single server.

Event Client Interface

EvClient interface enables:

1. Create and destroy EvClient objects.

2. Connect to and disconnect from event server.

3. Create and send event message to the server.

Event Server Interface

EvServer interface enables:

1. Create and destroy a single EvServer object.

2. Accept incoming EvClient connections.

3. Receive transmitted EVENTS from the clients.

4. Process received EVENTS (collecting, reformatting, logging, etc.).

5. Send formatted EVENTS via MQTT channel.

Events

Requirements

Events transmitted to the remote subscriber must be uniquely identifiable and well formatted. Each EVENT message must contain event CATEGORY and SOURCE (the origin of event) information. Event CATEGORY must define group, phase, and step for this event measures (e.g., DHCP client events), whereas SOURCE defines the process which owns and handles this event AND/OR network interface name this event belongs to.

When an event or event sequence is handled by multiple processes, the SOURCE is identified by common resource, this event refers to. Therefore the selection of interface name seems to be the right choice for many network communication-related interfaces, but also other SOURCES are allowed as well.

Events of one CATEGORY may create pairs (START / STOP), group sequences (START / STEP_X / STEP_Y / STOP) or be single events. Sequence information is coded in the SEQ field.

Each EVENT contains timestamp. This event was generated with a minimal resolution of 1 second, but higher 1 ms resolution is recommended.

TIME is expressed in UTC format to eliminate the need of time-zone adjustments or make it easy on receiver side.

If a node device does not have UTC time (e.g., at startup) the system time is used for reporting, starting from 00:00:00 (syslog convention).

Each EVENT may contain a message text field (MSG) to provide more details regarding the event. MSG is optional and can be empty when no extra information is added.

Format

EVENT object consists of following fields resulting from the above description:

In the transmitting channel, EVENTS are encoded in google protocol buffers format with Protobuf 2.0 syntax.

Protocol Buffers encoding are used to transmit EVENTS from EvClient to EvServer and from EvServer via MQTT to the remote subscriber.

event.proto

// List of event categories (proposal)
enum EvCategory
{
    BOOT = 0;           // target booting sequence until OpenSync is started
    UTC_TIME = 1;       // UTC time set (event message contains boot time stamp)
    WIFI_LINK = 2;      // WIFI link starting / ready / stopping / stopped
    GRE_LINK = 3;       // GRE link starting / ready / stopping / stopped
    WDS_LINK = 4;       // WDS link starting / ready / stopping / stopped
    DHCP_CLIENT = 5;    // DHCP client address assignment: starting / ready (addr leased) / stopped
    DHCP_SERVER = 6;    // DHCP server events: starting / ready (addr leased) / stopped
    REDIRECTOR = 7;     // Redirector link resolving / connecting / ready / stopped
    CONTROLLER = 8;     // Controller link resolving / connecting / ready / stopped
}
 
// Definition of event message
message Event
{
    required uint64 time = 1; // system time -> UTC time
    optional uint16 msec = 2; // milliseconds for improved time measurement accuracy
    required EvCategory cat = 3; // event category
    repeated string source = 4 [packed=true]; // event source, maybe defined by more than one value
    optional string seq = 5; // 
    optional string msg = 6;
}

Implementation

Implementation of this event-based time measurement system in OpenSync firmware software can be divided into four stages.

1. Event client shared library for EvClient objects with use of Protocol Buffers schema

2. Event server for EvServer object

3. MQTT transmitting channel logic

4. Generation of EVENTS in the embedded software with use of EvClient library

Event client library

1. Definition of Protocol Buffer schema for EVENT object / message

2. Simple API to create, destroy client objects, connect to event server, create and send events to the server

3. Client library internally take care of time stamp insertions, protobuf encapsulation and sending

4. Client object shall enable logging of EVENTS in the local logging system with OVSDB configuration

Event Server

Event server is enabled using an OVSDB configuration.

Event server utilizes the existing implementation of QM (Queue Manager), uses the existing QM Unix datagram socket to accept incoming connections from EvClients and to receive EVENT messages.

Event server shall implement time-based aggregation of receiving EVENT messages, i.e. collect the receivied data for a predefined time window before transmitting the data via MQTT channel.

Event server encapsulates the aggregated EVENT messages in packets with node device identification header containing HW and SW identification data.

MQTT Transmission

MQTT transmission link is already handled by QM. This MQTT link shall be used to add/register a new MQTT event topic channel to enable the subscription for.

Generation of Events

This part of implementation includes the following steps:

1. Identification of all events for time measurement with categorization, sequence and source definition

2. Updating of event category enumeration list in the protobuf schema

3. Identification of processes (OpenSync managers) using which the events are generated

4. Generation of events in selected processes using the EvClient usage pattern

EvClient Usage Pattern

1. Add initialization code for creation of global EvClient object connected to EvServer.

2. Insert events identified by Category, Source and Sequence in the proper places of code

Limited use of QM

The QM client-server protocol design is focused on peer-to-peer connection and uses stream sockets. This approach is not suitable for the “many-clients-to-one-server” communication topology. Datagram sockets are therefore used for this purpose instead.

Requirements

Enabled POSIX message queues in SDK:

CONFIG_POSIX_MQUEUE=y