Quick performance tests software¶

authors:	Joost Baars
date:	Mar 2020

Description¶

This page contains information about the quick performance tests that were implemented for the round trip and flood in C++. The goal of these measurements was to analyze the performance of different QoS policies in various configurations.

The software for the quick performance tests can be found in the directory src/demonstrators/RoundTrip/C++/PerformanceMeasurements/QuickQoSMeasurements/.

Dependencies¶

The dependencies necessary for succesful compilation of the quick performance measurements in C++ are the following:

CMake

C++17 compiler (Gcc 7+ for example)

Cyclone DDS library

Configuration¶

The performance tests implementation contains an implementation for the round trip application and one for the flood application. One of these implementations can be chosen before building the project.

The implementation can be configured by altering the following line in the CMakeLists.txt:

# Choose the wanted configuration
## Config parameter: can be configured for building the wanted performance test
## This can be "roundtrip" or "flood"
set(PROJECT roundtrip)

This line can be found at the top of the CMakeLists file. The roundtrip can be changed to flood for the flood implementation.

Building¶

The software can be build by executing the following commands in the command line (you should be in the main directory of the application! See Description for the directory)

mkdir build && cd build
cmake ..
make

These commands compile and create the executable for the performance measurements.

Execution¶

The application must be executed using various parameters. The application can be executed using the following command:

./<Application name> <device ID> <number of devices> <total messages / round trips> <filename for storing measurements>

<Application name>: The application you want to run, this can be performance_measurements_flood or performance_measurements_roundtrip
<device ID>: The ID of the device, must be unique, > 0 and <= total devices
<Total devices>: The amount of devices in the loop (the amount of applications running)
<Total msg / round trips>: For the flood application: Total messages send. For the round trip: Number of round trips to execute
<filename>: Optional, can be configured to change the default measurements file name. By default: measurements.csv.
The results are comma separated and appended to the file. Therefore, the measurements are not nicely formatted in the file.

The following example starts 4 nodes for the round trip. The slaves of the round trip are started in the background. Only the master is started in the foreground in this example. Each device pings a total of 1000 times. Therefore, there are 1000 round trips. The measurements of this test are stored in the file test1.csv

./performance_measurements_roundtrip 2 4 1000 & ./performance_measurements_roundtrip 3 4 1000 & ./performance_measurements_roundtrip 4 4 1000 &
./performance_measurements_roundtrip 1 4 1000 test1.csv

A description of the parameters can also be found when executing the application with no or incorrect parameters.

Note

Execution of the program

The round trip is initiated by the device with <device ID> = 1. Therefore, <device ID> = 1 should always be started after the other ID’s are started.

The devices added with <device ID> must be an increment of the previous one. This function does not dynamically search for the next device! So if there are three devices, device 2 and 3 should be started first. Afterwards, device 1 can be started.

The <total msg / round trips> parameter should be the same for each application.

Software¶

The application is written with customizability and ease of use in mind. Therefore, it is rather easy to create new tests. This chapter explains how tests can easily be added to the software. The round trip won’t be explained, because it is already explained in the Round trip in C++.

Class structure¶

The class structure is shown in the image below. The PerformanceTests class executes the wanted tests. In this class, the correct configuration can be set-up for the flood / round trip application.

The Flood / RoundTrip class is only locally created in the executeTest() function in the PerformanceTests class. This is done so the tests are performed in a clean Flood / RoundTrip object. The QoSManager object is given to the Flood / RoundTrip class as a pointer.

The Configurations class is a fully static class with static configuration functions inside. These static functions are configured in the PerformanceTests class and are executed in the Flood / RoundTrip class.

The performance measurements are only executed in the Flood / RoundTrip class. These results are requested by the PerformanceTests class (the executeTest() function) and stored in a file by the MeasurementsFile class.

Configurations¶

In the configurations class, new configurations regarding QoS policies can be added. They can be implemented with one configuration parameter.

The function must have the following structure in the header file:

static void <function name>(dds_qos_t *qos, dds_listener_t *listener, const int64_t configuration = 0);

This is a function structure that is used for every configuration. This makes it easy to select different configurations from the PerformanceTests class.

The <function name> tag should of course be changed to the wanted function name.

The C++ file must contain the configuration itself. This is an example of the reliability QoS policy in the Configurations class:

void Configurations::reliability(dds_qos_t *qos, dds_listener_t *listener, const int64_t configuration) {
   // DDS_RELIABILITY_BEST_EFFORT or DDS_RELIABILITY_RELIABLE
   dds_qset_reliability(qos, static_cast<dds_reliability_kind_t>(configuration), DDS_SECS(10));
}

The configuration is an int64_t because almost all parameters can be safely converted to this type. Because of this design choice, the configuration should first be converted to the correct enum type for the reliability.

Adding performance measurements¶

The QoS policies are configured using a QoSRoundtrip class. This class manages the default and custom QoS policies for each test.

A QoS policy can be configured using the following code:

QoSRoundtrip::QoSFunction configDefault;
configDefault(Configurations::reliability, DDS_RELIABILITY_BEST_EFFORT);

This is a struct containing the reliability test from the Configurations class with the configuration for that function. In this case, the configuration is DDS_RELIABILITY_BEST_EFFORT.

You can create as many of these configurations as you want.

The QoSRoundtrip class manages the QoS policies. More information can be found in Explanation of QoSRoundtrip. The configuration structures above should be configured with the QoSRoundtrip class.

Default configuration¶

If you want to change the configuration for a set of tests, the default configuration in the QoSRoundtrip class must be changed.

An example is given for a default test below. This executes the qosPolicyTests() with the reliability set to reliable and the durability to transient local.

These settings will be active with each test in the qosPolicyTests() except if the setting is overwritten by a custom configuration.

QoSRoundtrip::QoSFunction configDefault, configDefault2;

configDefault(Configurations::reliability, DDS_RELIABILITY_RELIABLE);
configDefault2(Configurations::durability, DDS_DURABILITY_TRANSIENT_LOCAL);

QoSRoundtrip qosConfig; // You must have a qosConfig object
qosConfig.configureDefaultQoS(configDefault, configDefault2);
qosPolicyTests();

The function qosConfig.configureDefaultQoS() accepts as many configurations as you want.

The default configuration can also be reset to the default configuration. This can be done by the following command:

qosConfig.resetDefaultQoS();

Custom Configurations¶

If you want a configuration to be executed once. You should alter the custom configuration in the QoSRoundtrip class.

This next codeblock shows how custom configurations are made. These custom configurations are the “master” configuration. This overwrites the default configuration if they alter the same QoS policy.

The next codeblock assumes that you already have made a QoSRoundtrip object with the name: qosConfig.

QoSRoundtrip::QoSFunction config;
config(Configurations::deadline, DDS_SECS(10));
qosConfig.configure(config);
executeTest("deadline", &_qosConfig);

The codeblock above configures the deadline QoS policy with a configuration parameter of DDS_SECS(10).

The qosConfig object is configured normally (without using configureDefaultQoS()). The `QoSRoundtrip object containing the wanted QoS policies is given to the executeTest() function with a custom name.

The executeTest() function needs a string containing the name of the test and the QoSRoundtrip object. This function executes the round trip and stores the result in a measurements file.

Explanation of QoSRoundtrip¶

The QoSRoundtrip class controls the QoS policies for the round trip performance measurements. This class contains all the necessary QoS objects.

The class contains the dds_qos_t * for the writer, reader, topic of the writer and the topic of the reader. Additionally, it contains the dds_listener_t * for the listener. Each of these has a custom configuration that is used for the performance tests. As well as a default configuration that will be copied into the custom configuration before the custom QoS policies are set-up.

This class also contains the struct that manages the different configurations. This struct is called QoSFunction.

This struct contains a function containing a QoS policy. It must be a function with the format:

void <function name>(dds_qos_t *qos, dds_listener_t *listener, const int64_t configuration);

This struct also contains for what object it is meant. It can be configured that the QoS policy is only configured for the writer, or for the reader. The applicable() function can be used to set for whom the QoS policies must be configured.

// This function is part of the QoSFunction struct
void applicable(const bool writer, const bool reader, const bool writeTopic, const bool readTopic);

The following functions can be executed for configuring the QoS policies:

// Resets the default QoS policies in the class
void resetDefaultQoS();

// Configures the default QoS policies.
void configureDefaultQoS(ConfigFunctions... configFunctions);

// Configures the custom QoS policies
void configure(ConfigFunctions... configFunctions);

The ConfigFunctions must be of type QoSFunctions.

The configure() function overwrites the previously called configure() function. The configureDefaultQoS() function does not overwrite the previously called configureDefaultQoS() function. This can only be cleared using the resetDefaultQoS() function.

Measurements¶

The measurements of each test are appended to a CSV file. This is not a human-readable structure. Therefore, this chapter explains how to make the results readable relatively fast.

Open the measurements file with Excel. Column A should contain the measurement names and column B the measurements. The current test executes 39 QoS tests with 8 different default configurations. Therefore, a table must be created with 39 rows and 8 columns.

Place this command in an empty cell: =INDEX($B:$B,ROW(B1)+((COLUMN(B1)-2)*39)).

Then select the square on the right bottom of that cell and select 39 rows and 8 columns with it.