Execute tests in parallel

In this section of the guide, we compare the virtual platform tests and hardware tests, and explain how to run tests in parallel to further increase test throughput.

In Build and run unit tests, we saw that running 124 unit tests on the two platforms took this amount of time on the an example system:

  • Arch Pro: 488 seconds
  • MPS2+ FVP: 259 seconds

These runs both test the same functionality. Because Arm virtual platforms are 100% functionally accurate, the total test time is cut almost in half while losing no information. The speed increase when using virtual platforms compared to hardware can be attributed to several factors, including:

  • No flash time
  • A faster clock speed on virtual platforms that run on a host OS than a development board

To speed up regression tests even more, you can run virtual platforms in parallel and reduce the testing time. This is another substantial benefit of using virtual platforms for running tests. Scaling with more simulations in parallel to speed up and run more tests is simple and easily maintained as opposed to buying and managing more hardware boards. Running simulations in parallel with Mbed OS is straightforward. You add the --parallel switch with the number of runs to perform in parallel.

For example, to run two tests in parallel on the previous test suite, use the following command:

mbedgt --fm FVP_MPS2_M3:DEFAULT -n features-device_key*,features-frameworks*,features-storage*,tests-integration*,tests-mbed_platform* --parallel 2

Here are the results of the Arch Pro, MPS2+ FVP, and MPS2+ FVP running two tests in parallel:

Platform # Tests Time (seconds)
% Speed increase from Arch Pro
Arch Pro Dev Board
124
488
0%
MPS2+ FVP
124
259
88%
MPS2+ FVP (2 in parallel)
124
160
205%

The run time for a regression test suite decreases by half for each simulation running in parallel. This is limited by the computation power of the host machine and the size of the test suite. Running three tests in parallel results in a speed increase of approximately 300%, running four tests in parallel results in a speed increase of approximately 400%, and so on. Virtual platforms enable this regression suite execution speed increase to most embedded software development testing infrastructures, without sacrificing functional accuracy.

Previous Next