Matrix Board Communication in C

authors:Stefanos Florescu
date:March 2020

Before following this guide in order to execute the matrix board communication make sure you have successfully installed CycloneDDS (Setup Guide for CycloneDDS).

Building the executable

Make sure your local repo is up to date with the HighTech-nl repo and move into the directory /dds-demonstrators/src/demonstrators/MatrixBoard/C:

$ mkdir build
$ cd build
$ cmake ..
$ make

Running the Matrix Board Communication

The demonstrator can be executed either on one machine by creating different processes (in different terminal windows) or by creating instances of the matrix boards on different nodes (i.e. on Raspberry Pis). A combination of the two methods can also be utilized, in other words running multiple processes on each of the nodes in the network.

In order to run a matrix board instance execute the following command by providing an integer number which is the ID of that particular board (i.e. a positive number):

$ ./MBC <MBC_ID>

Note

In the case that the wrong network interface is chosen by DDS at startup of the application see the notes in Setup Guide for CycloneDDS for details on how to utilize the desired network interface.

Note

See Setting up Raspberry Pi network for a complete guide on Raspberry Pi networking as well as the IP addresses for the network that is already in place.

Running the Matrix Board Communication in a Docker Container

Similarly to what we described above, the MBC experiment can be executed from a Docker Container. First make sure you follow the guide on how to build the Docker Image which can be found in the Docker Guide: How to create and run a Docker Image under “Building Demo Image”. First create a new container using:

$ docker run -it --network="host" --name <container name> <image name>

You can open another terminal/Powershell and have as many processes on that container as you want using per terminal:

$ docker exec -it <existing container name> bash

Navigate into the MatrixBoard/ directory and run one of the three MBCs using:

$ ./MBC_poc<1 or 2 or 3> <MBC_ID>

A demo with two different containers can be seen in the video below. The terminals on the left side are processes of container named dds1 while the ones on the right belong to container dds2. Then we change the network interface so that it works properly on Windows (more details on this can be found in the warning right below). Finally, we start Matrix Board with ID 1 and 3 in container dds1 and Matrix Board with ID 2 and 4 in dds2. This configuration demonstrates the inter-container communication as well as the communication within the same container.

Warning

Due to the networking issue of the Docker Desktop (see in Docker Guide: How to create and run a Docker Image under “Running Docker Containers” for more info) on Windows/MacOS we need to change the DDS networking interface in the docker_cyclonedds_config.xml in /root/. In the video below you can find an example of how to do that. On a Linux machine this is not required and your container will communicate over the network.

Matrix Board Communication: Proof of Concept 1 (poc1)

The first version of the Matrix Board Communication (MBC) was a simple implementation that has as a goal to introduce the user to a dynamic application of the DDS protocol. Each matrix creates the following entities:

  • 1 Participant on the default domain.

  • 1 Write topic, used to publish the traffic recorded by that particular MBC.

  • 1 Writer, used to publish the traffic of that particular MBC.

  • Array of read topics, starting from the ID of that particular MBC and up to a
    predefined constant number.
  • Array of readers, starting from the ID of that particular MBC and up to a
    predefined constant number.

In order to achieve the dynamic switching of boards, we create an array of topics and an array of readers on all those topics up to a predefined number (MAX_SEARCH). As a result, the dynamic behavior is achieved in a purely algorithmic method, by simply iterating up to that MAX_SEARCH until we find the two closest active MBCs. Furthermore, each MBC continuously writes its own topic without checking if there are any active readers. This method, albeit simple, will not have great performance especially due to the following reasons:

  1. Large switching time, especially for large gaps between MBCs (i.e. between board
    with ID 1 and board 9000).
  2. Increased overhead in each polling iteration, since the closest MBCs are always
    recalculated.
  3. Increased algorithmic complexity, due to two nested for loops used for the
    dynamic search algorithm.
  4. Unnecessary network usage, since in every polling iteration we write the traffic
    without checking for events (i.e. new data being available and/or active readers on that topic).
  5. Unnecessary memory usage, since we only require at most 2 read topics and 2
    readers per MBC and we actually have two arrays of size MAX_SEARCH - MBC_ID
    with signed 32-bit integers at each entry.

Matrix Board Communication: Proof of Concept 2 (poc2)

The second version of the Matrix Board Communication (MBC) has as a goal to expand upon poc1 while solving some of the issues of poc1 that were presented previously. This solution is based on the use of status and event driven DDS implementation. Similarly to poc1, each matrix creates the following entities:

  • 1 Participant on the default domain.

  • 1 Write topic, used to publish the traffic recorded by that particular MBC.

  • 1 Writer, used to publish the traffic of that particular MBC.

  • Array of read topics, starting from the ID of that particular MBC and up to a
    predefined constant number.
  • Array of readers, starting from the ID of that particular MBC and up to a predefined
    constant number.

The main search algorithm remains the same, but now each MBC writes to its own topic only when there are matched active MBCs that should read that topic. This is achieved by using on the writer side the dds_get_publication_matched_status function which returns a struct containing information on the number of matched readers, the total number of matched readers, the change of readers and the handle of the last matched reader (more information on this function can be found Statuses for DataWriters). Similarly on the reader side we introduce the function dds_get_subscription_matched_status, which provides the same information about matched writers to that particular reader (more information on this function can be found Statuses for DataWriters). As a result, we can now see if we have an active writer to a topic without having to do a dds_take or dds_read to see whether there are any available samples on that topic. Although, these additions do not fix all of the performance issues introduced with poc1, the list of performance bottlenecks is reduced to the following:

  1. Lower than poc1 but still large switching time, especially for large gaps between
    MBCs (i.e. between board with ID 1 and board 9000).
  2. Reduced overhead by comparison to poc1 but still increased overhead in each polling
    iteration, since the closest MBCs are always recalculated.
  3. Increased algorithmic complexity, due to two nested for loops used for the dynamic
    search algorithm.
  4. Unnecessary memory usage, since we only require at most 2 read topics and 2 readers
    per MBC and we actually have two arrays of size MAX_SEARCH - MBC_ID with signed
    32-bit integers at each entry.

Matrix Board Communication: Proof of Concept 3 (poc3)

The third version of the Matrix Board Communication (MBC) is the most complex and has as a goal to solve all of the previously introduced issues. The approach in this case is completely new by introducing the following novelties:

  • Membership, each MBC will have to announce its liveliness by writing to a dedicated topic.

  • Liveliness changes, using listeners attached to callback functions
    (more information can be found Types of Listeners).

  • Usage of sample_info to extract valuable sample information.

  • Hash tables, each MBC will have a hash table where each active MBC will be stored.

Each MBC will have the following entities:

  • 1 Participant on the default domain.
  • 1 Write topic, used to publish the traffic recorded by that particular MBC.
  • 1 Writer, used to publish the traffic of that particular MBC.
  • 2 Readers, used to read the traffic published by other MBCs.
  • 1 Membership topic, where the liveliness of all active MBCs can be found.
  • 1 Membership writer, used to announce the liveliness of that particular MBC.
  • 1 Membership reader, used to read the membership topic.

In a nutshell, a new active MBC will announce that it is alive by writing to the membership topic its own ID number, all of the other MBCs will detect this liveliness change using their liveliness change callback function and will read the membership topic. Upon reading the ID of the new MBC each of the previously active MBCs will add that new MBC in their hash table using the modulo of the publication handle of that particular sample over the predefined hash table size as the hash code. Furthermore, after adding the new MBC all of the previously active MBCs will also write to the membership topic their own IDs. Therefore, the new MBC will add all of the pre existing MBCs to its own hash table. Similarly, when a MBC goes offline, the other MBCs detect that event through the same liveliness change callback function and extract the publication handle of the offline MBC using the function dds_get_liveliness_changed_status which returns a similar struct to the functions presented in poc2. Using that handle each of the online MBCs removes the offline MBC from their hash table.

At this point it is interesting to present some of the functions the hash table utilizes for all of the operations mentioned above.

//used to calculate the hash code.
dds_instance_handle_t hashCode(dds_instance_handle_t key);

//search for a MBC based on its handle.
struct DataItem* hash_search(dds_instance_handle_t key);

//insert a new MBC based on its handle.
void hash_insert(dds_instance_handle_t key, int32_t data);

//delete a MBC based on its handle.
struct DataItem* hash_delete(struct DataItem* item);

//print all contents of the hash table (i.e. mainly for debugging).
void hash_display();

//count the number of entries in the hash table.
int hash_count();

//find the smallest ID number in the hash table (excluding the ID of the MBC requesting this).
int hash_return_first_min(int32_t mbc_id);

//find the second smallest ID number in the hash table (excluding the ID of the MBC requesting this).
int hash_return_second_min(int32_t mbc_id, int32_t min1);

Using the two last functions from the code block above, each MBC can determine to which other MBCs it has to subscribe in order to receive their traffic values. This implementation solves the algorithmic complexity issue since the hash table finds entries based on their hash and not by looping over all of them. Furthermore, memory is used efficiently without keeping unnecessary entities and hash entries. Overall, this implementation minimizes the time required by the dynamic subscription and outperforms the previous proofs of concept by a vast margin.