Command-Line Interface

The SPBench framework works with a command-line interface (CLI).

The CLI offers some commands. You can check them running:

./spbench --help

For each command there is also a “help” guide. You can access it using:

./spbench [command] -h

SPBench install

./spbench install -h

Description:

It install the dependencies for the SPBench benchmarks. See installation for more information.

Optional arguments:
  • -app <some_application>

    Install dependencies only for the seleted application.

Global configuration file

./spbench global-config -h

Description:

It opens a JSON configuration file for adding global compiling dependencies. The information added to keys in the global configuration file will override the information present in the individual configuration file of each benchmark when generating makefiles. The ‘EXTRA’ keys do not override the original information, but append to it instead. See globalconfig for more details.

Optional arguments:
  • -editor <editor_name> (Default: nano, vim, or vi)

    You can select a different text editor available in your machine.

Usage example:

  • ./spbench global-config -editor <text_editor_name>

Operations with benchmarks

Update benchmarks

Description:

It updates the Makefiles with new compiling configurations or create a new Makefile if there is no one.

You should run it every time you change the SPBench path or SPBench version, after a SPBench update, or after running the configure command or any similar case. If in doubt, run it!

Note

Do not forget to recompile the benchmarks for the command update to take effect.

Usage examples:

For all applications:

  • ./spbench update

For a specific benchmark:

  • ./spbench update -bench <benchmark_name>

For all benchmarks based on a specific application:

  • ./spbench update -app <application_name>

For all benchmarks implemented with a specific PPI:

  • ./spbench update -ppi <ppi_name>

For more info:

  • ./spbench update -h

Add a new benchmark

./spbench new -h

Description:

It allows for creating a new benchmark based on a SPBench application. This command will create a fresh copy of the selected application and add it to the database.

Required arguments:
  • -bench <new_benchmark_id>

    Set a name for your new benchmark.

  • -app <application_name>

    You must insert an application ID. See applications for more details or run [./spbench list-apps] to see the available applications.

Optional arguments:
  • -ppi <any_ppi>

    We recommend inserting a name for the PPI you will use here. But you are free to insert any identification word. Default is ‘sequential’.

  • -from <other_benchmark_id>

    Use this option to create your new benchmark as a copy of an existent benchmark. Insert the name of the benchmark you want to copy from

  • -nsources

    Use this option to create a new benchmark with support for multiple sources

Usage example:

  • ./spbench new -bench <new_bench_name> -app <application_name> -ppi <ppi_name>

Edit a benchmark

./spbench edit -h

Description:

It opens a choosen benchmark for editing the source code.

Required arguments:

  • -bench <benchmark_id>

Optional arguments:
  • -editor <editor_name> (Default: nano)

    You can select a different text editor available in your machine.

Usage example:

  • ./spbench edit -bench <benchmark_name> -editor <text_editor_name>

Configure a benchmark

./spbench configure -h

Description:

It allows users to add compiling dependencies for a benchmark. It opens a JSON configuration file for users to write the required dependencies. See Compiling Configuration for more details.

Required arguments:

  • -bench <benchmark_id>

Optional arguments:
  • -editor <editor_name> (Default: nano)

    You can select a different text editor available in your machine.

Usage example:

  • ./spbench configure -bench <benchmark_name> -editor <text_editor_name>

Compile benchmarks

./spbench compile -h

Description:

It allows users to compile the benchmarks.

Required arguments:
  • -bench <benchmark_id> or -ppi <ppi_id> or -app <application_id>

    Name ID of the benchmark(s) to compile. These arguments can be combined. Example:

    • -bench my_awesome_bench compiles the benchmark called ‘my_awesome_bench’.

    • -bench all compiles all benchmarks.

    • -app ferret compiles all ferret benchmarks.

    • -ppi tbb compiles all benchmarks parallelized with TBB.

    • -app ferret -ppi tbb compiles all ferret benchmarks parallelized with TBB.

Optional arguments:
  • -editor <editor_name> (Default: nano)

    You can select a different text editor available in your machine.

  • -clean

    Removes all the object files that had been created for the selected benchmark(s) before compile it.

Usage example:

  • ./spbench compile -bench <benchmark_id> -editor <text_editor_name>

Execute benchmarks

./spbench exec -h

Description:

This command runs the given benchmarks.

Required arguments:
  • -bench <benchmark_id> or -ppi <ppi_id> or -app <application_id>

    Name ID of the benchmark(s) to execute. These arguments can be combined. Example:

    • -bench my_awesome_bench executes the benchmark called ‘my_awesome_bench’.

    • -bench all executes all benchmarks.

    • -app ferret executes all ferret benchmarks.

    • -ppi tbb executes all benchmarks parallelized with TBB.

    • -app ferret -ppi tbb executes all ferret benchmarks parallelized with TBB.

  • -input <input_id>

    Insert an input ID (run ./spbench list-inputs). You can also insert multiple inputs for multiple source benchmarks.

Optional arguments:

Warning

  • -debug

    By default, SPBench captures the benchmark output while it is running and prints all the output at once at the end of the execution. SPBench does this so that it can compute some additional metrics based on the output of the benchmarks. If you want it to print the output on-the-fly (e.g. to print debug info added by you, or any print message), you must use the -debug argument in the execution command. Notice that, if this option is enabled, some advanced SPBench features may not work properly. It will not show the standard deviation of the metrics or store the result in the general log file, for instance.

  • -nthreads <number_of_threads>

    Number of threads. It can be used inside the benchmarks as a global variable. You can insert a single number or a range using the ‘:’ character (e.g. -nthreads 1:10). You can also define iteration step of the range (e.g. -nthreads 4:4:16 would run the benchmarks with 4, 8, 12, and 16 threads). Default value: 1.

  • -batch <batch_size>

    It can be used to change the batch size (default: 1).

  • -batch-interval <time_interval_sec>

    It can be used to change the batch time interval. Default = 0 (disabled).

  • -frequency <items_per_second>

    A limited number of items will be available for the source per second. By default, no limit is aplied, which is equivalent to maximum frequency. The maximum frequency is defined by your system’s memory or disk access speed.

  • -freq-pattern <frequency_pattern>

    (pattern,period,max,min). This argument overwrites the -frequency command and sets a varying frequency for the input stream. For this argument users need to supply a tuple string: <pattern,period,max,min>, where pattern can be “wave”, “spike”, “binary”, “increasing” or “decreasing”. Period is the duration in seconds of one cycle of the pattern. And min is the minimum frequency and max the maximum frequency, defined in items per second. For the spike pattern users can also set the duration of the spikes as percentage of the period (0-100) (e.g. <pattern,period,max,min,spike>). The default spike duration is 10 percent of the period. Usage example: -freq-pattern wave,10,5,80

  • -in-memory

    It runs the application in in-memory mode.

  • -latency

    Print average latency results at the end of the execution.

  • -throughput

    Print the average throughput at the end of the execution.

  • -resource-usage

    It prints the global memory and CPU usage at the end of the execution. To use it is required to run the benchmark as root or adjust paranoid value.

  • -monitor <time_interval> (millisecond)

    Use this option to monitor latency, throughput, CPU and memory usage. You must indicate a monitoring time interval in milliseconds (e.g. 250).

  • -monitor-thread <time_interval> (millisecond)

    Use this option to monitor latency, throughput, CPU and memory usage. It works the same as the -monitor command above, but it runs the monitoring computations in an individual thread. It may be used for decreased performance overhead in the application when monitoring with low time intervals.

  • -latency-monitor

    It monitors latency per stage and per item and generates a log file.

  • -user-arg <user_custom_argument>

    User custom argument. Allow users to pass one or more arguments to use inside the benchmarks. Single argument: [-user-arg single_argument] or [-user-arg “single argument”]. Multiple arguments: [-user-arg two arguments] or [-user-arg two -user-args arguments].

  • -test-result

    This option will run a correctness test on the output (when available) after the benchmark execution.

  • -repeat <number_of_repetitions>

    Use this argument to repeat this execution n times. It will compute and show a summary of the selected metrics at the end.

  • -print

    When this parameter is added, the exec command will only print on the screen the execution command it would use to run the benchmark. You may use it to manually change some specific aspect of the execution that is not supported by SPBench.

  • -quiet

    Run it in quiet mode. If any metrics are selected, it shows the results in a short format.

  • executor <executor_name_and_parameters>

    You can use this argument to change the way the benchmarks are executed. Usage example: ./spbench exec -executor "-my_executor my_executor_parameters" -bench ...

  • -d (Bzip2 only)

    Enables decompress mode for Bzip2 benchmarks.

Usage example:

  • ./spbench exec -bench ferret_tbb_farm -input small -nthreads 8

    Simple run.

  • ./spbench exec -bench lane_sequential -input medium -latency -throughput -resource-usage -test-result -repeat 5

    Running with basic metrics, result testing, and repeating the execution 5 times.

  • ./spbench exec -bench ferret_grppi_farm -input test -latency -throughput -nthreads 8:2:16 -user-arg omp

    Running a Ferret GrPPI benchmark with 8, 10, 12, 14, and 16 threads, and selecting the OpenMP backend through a custom user argument.

  • ./spbench exec -bench ferret_sequential -input test -executor "gdb"

    Running a benchmark with GDB debugger as executor.

  • ./spbench exec -bench bzip2_ff_farm -d -input large -nth 6 -latency-monitor -monitor 300

    Running Bzip2 in decompress mode with monitoring metrics enabled.

  • ./spbench exec -bench person_seq_ns -input large -input my_custom_input -in-memory

    Running multiple source Person Recognition benchmark with two inputs and in-memory enabled.

  • ./spbench exec -bench my_custom_bench -input small -batch 4 -frequency 50 -user-arg "My awesome argument"

    Running a custom benchmark with batch size = 4 and maximum of 50 input items per second, plus a custom argument.

  • ./spbench exec -bench my_custom_bench -input huge -batch-time 1.2 -freq-patt binary,7,10.5,70.8

    Running a custom benchmark with the workload class huge, batch time interval = 1.2 sec., and a binary frequency pattern that oscilates the frequency between 10.5 and 70.8 every 7 seconds.

Tip

If you use repetitions (-repeat <n>) and -nthreads <start:step:end> in range mode together in the exec command, a specific resulting log will be automatically generated inside the log folder. This log file will store average latency, throughput, and exec. time, with their respective std. deviation side by side. It may be handy for running experiments and saving time gathering resulting data. The resulting log file is ready for users to draw charts over the results.

Note

See How to run SPBench, Management Options, and Performance Metrics for more helpful information.

Usage example:
  • ./spbench exec -bench ferret_ff_farm -input test -latency -throughput -nthreads 1:2:10 -repeat 3

    You can find the resulting log file for this usage case in the log/ferret_ff_farm_1-2-10.dat file.

List benchmarks

./spbench list -h

Description:

This command lists all SPBench benchmarks, including the user’s custom benchmarks.

Optional arguments:
  • -app <application_name>

    Lists only benchmarks based on the given application.

Usage example:

  • ./spbench list

Delete a benchmark

./spbench delete -h

Description:

It completely deletes a given benchmark.

Required arguments:
  • -bench <benchmark_id>

    Name of the benchmark to delete.

Usage example:

  • ./spbench delete -bench my_not_so_awesome_bench

Rename a benchmark

./spbench rename -h

Description:

This command can be used to change the name of a given benchmark inside SPBench.

Required arguments:
  • -old-name <old_bench_id>

    Insert the old and existent benchmark ID.

  • -new-name <new_bench_id>

    Insert the new ID.

Usage example:

  • ./spbench -old my_certainly_working_bench -new my_doubtfully_working_bench

Clean a benchmark

./spbench clean -h

Description:

Clean a benchmark (similar to make clean). It also removes old operators.

Required arguments (at least one of them must be used):
  • -benchmark <benchmark_id>

    Insert the name of a benchmark to clean (similar to make clean). You can use ‘all’ to clean all benchmarks.

  • -app <app_id>

    Clean all benchmarks based on a specific given application.

  • -ppi <ppi_id>

    Clean all benchmarks implemented using a specific given PPI.

  • -logs <ppi_id>

    Delete all execution logs of SPBench.

  • -outputs <ppi_id>

    Delete all the outputs generated by the benchmarks

Usage example:
  • ./spbench clean -bench all

    Clean all benchmarks.

  • ./spbench clean -app lane_detection -logs

    Clean all lane detection benchmarks and all SPBench logs

  • ./spbench clean -ppi <my_awesome_ppi> -outputs

    Clean all benchmarks using a specific PPI and the outputs from all benchmarks.

  • ./spbench clean -logs -outputs

    Delete all SPBench logs and the outputs from all benchmarks.

Operations with input workloads

Download input workloads

./spbench download-inputs -h

Description:

Download the inputs for the SPBench benchmarks.

Optional arguments:
  • -app <application_name>

    Use this option for downloading inputs for a specific application.

  • -class <workload_class>

    You can insert the class of workload you want to download (leave it empty to download all classes). Possible options: all, test, small, medium, large, and huge.

  • -force

    Use this option to delete any respective input file and download it again.

Usage example:
  • ./spbench download-inputs (recommended)

    It will download everything.

  • ./spbench download-inputs -app bzip2 -class small -force

    Bzip2 has some large files to download, so users can use this option to avoid downloading everything.

List inputs

./spbench list-inputs -h

Description:

This command lists all application’s inputs provided by SPBench, including users custom ones.

Optional arguments:
  • -app <application_name>

    Lists only inputs based on the given application.

Usage example:

  • ./spbench list-inputs

Add new input ID

./spbench new-input -h

Description:

It allows for adding an alias for a custom input. The chosen alias will be registered for future uses. You must use this option to run the benchmarks using other workloads than the ones provided by SPBench.

Required arguments:
  • -id <new_input_id>

    Set a name for your new input (existent entries will be replaced)

  • -app <application_name>

    Insert an application from the SPBench suite.

  • -input "<input_string>"

    You must insert the input string (e.g., “/home/user/my_awesome_input_file …”)

Optional arguments:
  • -md5 <md5_hash_for_result_testing>

    SPBench uses the associated md5 hash to check the correctness of the result for this input (Optional). ATTENTION: do not insert the input file resulting md5 hash here. You must insert the md5 hash regarding the resulting output file. We strongly recommend that you run your new input with a sequential benchmark to compute this hash of the resulting output file.

Usage example:

  • ./spbench new-input -id immensely_huge -app bzip2 -input "/home/me/awesome_songs/never_gonna_give_you_up-remix.mp3"

Delete a input ID

./spbench delete-input -h

Description:

Delete a given input ID from the SPBench inputs registry. It does not delete the associated files.

Required arguments:
  • -id <new_input_id>

    Insert the ID of the input to delete.

  • -app <application_name>

    Insert the associated app for the given input.

Usage example:

  • ./spbench delete-input -id my_super_extra_medium_input -app lane_detection

Operations with applications

List applications

./spbench list-apps -h

Description:

List the information of all available applications in SPBench.

Usage example:

  • ./spbench list-apps

Add a new application

./spbench -new-app -h

Description:

This command can be used for adding a new application to SPBench. This command will create a toy application and add it to the database. Users can use this toy application as a template to write their onw application or import/translate into SPBench an existing one. Please, check Adding New Applications for additional instructions.

Required arguments:
  • -app <application_name>

    You must insert an ID for your new application.

  • -operators [list of operators]

    You must insert a list with the names of the operators of your new application (in order). Source and Sink operators are automatically included. You should not include them.

Optional arguments:
  • -name <"application name">

    By default, the application’s name would be the same as the application ID. However, you can define a different name for it using this argument.

  • -description <"description">

    A description of your application.

  • -notes <"notes">

    Any notes, observations, or usage instructions for your application.

Usage example:

  • ./spbench new-app -app my_new_app -name "Full Name of My App" -op anOperator anotherOp lastOp -notes "Item order must be guaranteed on the output"

Delete a application

./spbench delete-app -h

Description:

It completely deletes a given application.

Required arguments:
  • -app <application_id>

    Name of the application to delete.

Usage example:

  • ./spbench delete-app -app my_test_application

Operations with operators

List operators

./spbench edit-op -h

Description:

List all available operators for a given benchmark.

Required arguments:
  • -benchmark <benchmark_id>

    You must give the name of the respective benchmark you want to see the operators list (mandatory).

Usage example:

  • ./spbench list-op -bench my_awesome_bench

Edit an operator

./spbench edit-op -h

Description:

Open an operator of a given benchmark to edit its source code.

Required arguments:
  • -benchmark <benchmark_id>

    Enter the name of an application benchmark to edit (mandatory)

  • -operator <operator_id>

    Enter the name of the benchmark’s operator to edit (mandatory)

Optional arguments:
  • -editor <editor_name>

    Enter the name of a text editor available in your machine, e.g. “vim”, “nano”, etc. (default: nano)

Usage example:

  • ./spbench edit-op -bench ferret_sequential -op segmentation -editor vi

Reset operators

./spbench reset-op -h

Description:

Reset to default the source code of all operators for a given benchmark (it still keeps the old operators as backup until this command is executed again).

Required arguments (at least one of them must be used):
  • -benchmark <benchmark_id>

    Insert the name of a benchmark to reset its operators. Caution: It will remove any current modifications on the operators. You can use [-bench all] to reset the operators for all benchmarks.

  • -app <app_id>

    Reset operators for all benchmarks based on a specific given application.

  • -ppi <ppi_id>

    Reset operators for all benchmarks implemented using a specific given PPI.

Usage example:

  • ./spbench reset-op -bench my_awesome_bench