Performance Metrics

In SPBench users can evaluate performance in two ways. One way is to select a set of specific metrics before the benchmark is run. In short, four metrics can be collected by the SPBench from the execution of a benchmark:

• Latency

• Throughput

• CPU usage (%)

• Memory usage (KB)

These metrics can be collected at different levels, such as per time unit, per operator, per source, or global average. Some of them can be combined as well.

Also, the metrics to be collected with SPBench can be defined either statically (selected and configured before starting the run), or dynamically (collected at any point during the benchmark run).

The table bellow shows the details.

Metric	Granularity	Usage
Latency	Global average (end-to-end) Global average (per-operator) Per time interval Per item	Dynamic and static
Throughput	Global averag Average per time interval	Dynamic and static
CPU and Mem. Usage	Global average Per time interval	Static

Statically Selecting Performance Metrics

Statically selected metrics are the metrics that are selected as an argument through the exec command in the CLI. See Command-Line Interface or run ./SPBench exec -h for more information.

Here are five optional arguments for metrics you can use when running the ‘exec’ command:

• -monitor <time_interval>

  It monitors latency, throughput, CPU and memory usage. Here you can indicate a monitoring time interval in milliseconds (e.g. 250.Optional). It will generate a log file inside a log folder (spbench/log/). This file will contain the results got for each metric at each time interval.

• -latency

  It prints the average latency results for each application operator on the screen after the execution.

• -throughput

  It prints the average throughput results after the execution.

• -latency-monitor

  It monitors the latency per stage and per item and generates a log file inside the log folder.

• -resource-usage

  It prints the global memory and CPU usage for the selected application.

Note

You must run the benchmark as root or adjust paranoid value to use the resource usage metric.

Dinamically Selecting Performance Metrics

SPBench also allows some metrics to be collected at runtime, such as latency and throughput.

Latency can be returned in two ways. The first is the current latency, which is the latency of the last item processed in the sink. Another option is to get the average latency of the items that have been processed so far.

To get the current latency: spb::SPBench::getCurrentLatencyMs()

To get the average latency: spb::SPBench::getAverageLatencyMs()

To get throughput (items per second): spb::SPBench::getItemsPerSec().