A funny thing happened yesterday: I've just merged my ranch with code coverage reports for my project with PlatformIO, opened Twitter to tweet about it, and precisely at that moment, PlatformIO themselves posted their own blog post!
Of course, I found this coincidence exciting and decided to tell my story with an accent on automated unit testing and code coverage reports with GitHub Actions.
This article is written as a result of SenseShift project development. If you are a fan of Virtual Reality and Open Source, consider checking it out: SenseShift is an open-sourced firmware for the VR Accessories ecosystem. With SenseShift you can build yourself VR Haptic Vests, Haptic Gloves, and much more!
Preparing the project
Since we will run our tests and collect their coverage in GitHub actions, we are limited to the Native platform and Local Test Runner, and we need to correctly structurize and organize the project by splitting the source code into multiple components and placing them into the lib_dir. Otherwise, tests would not compile since the Arduino library is incompatible with the Native platform.
I'm aware of the ArduinoFake library existence, but it is not the topic of this article.
It took me quite some time to correctly prepare my project, but in the end, I successfully decoupled abstract "core" methods and classes from hardware-dependant ones.
Once I've successfully run tests locally, it is time to collect the baseline for future coverage reports.
Collecting initial coverage for the firmware
To correctly calculate the coverage percentage for the project, we need to collect initial coverage for the firmware. It is essential because we need to compare test coverage with files, classes, and functions that are actually used. We can collect initial coverage by compiling binaries with the following build flags:
Compiling a project with these flags will generate
.gcno files containing information about which source code lines were used for project compilation.
However, in my case, a compilation failed with the
region 'dram0_0_seg' overflowed by 64368 bytes error. A simple workaround is available to prevent this issue: by modifying the line
len = 0x2c200 - 0xdb5c in the
~/.platformio/packages/framework-arduinoespressif32/tools/sdk/esp32/ld/memory.ld file to
len = 289888, we can force firmware to compile successfully.
sed -i "s/len\s=\s0x2c200\s-\s0xdb5c/len = 289888/" ~/.platformio/packages/framework-arduinoespressif32/tools/sdk/esp32/ld/memory.ld
However, it will not work on the actual MCU, and it even can damage the hardware! Do not try to flash the firmware compiled with these changes!
Do not try to flash firmware with modifyed memory segments!
lcov file for initial coverage
The initial lcov.info file is usually generated with the
lcov -i -d ./build/binaries -c -o ./build/lcov/lcov.info.initial command: adding an
-i flag marks it as "initial" and does not require
gcda files. But for the Espressif compiler, we need to slightly modify it by adding a custom
pio run -e default lcov -i -d ./.pio/build/default/ -c -o ./build/lcov/lcov.info.initial -gcov-tool ~/.platformio/packages/toolchain-xtensa-esp32/bin/xtensa-esp32-elf-gcov
Running tests and collecting covered lines
We have already added
-lgcov --coverage flags previously, so compiling tests will generate
gcno files, and running them will produce
gcda files. Now we need to process them and generate
lcov.info files by running the same command as before, but now without
pio test -e native lcov -d ./.pio/build/native/ -c -o ./build/lcov/lcov.info.test
Merging and cleaning up results
Now that we finally have both initial and test coverage, it is time to merge them to calculate final coverage:
# Merge coverage lcov -o ./build/lcov/lcov.info -a ./build/lcov/lcov.info.initial -a ./build/lcov/lcov.info.test # Clean up libraries lcov --remove ./build/lcov/lcov.info '/usr/include/*' '*.platformio/*' '*/.pio/*' '*/tool-unity/*' '*/test/*' '*/MockArduino/*' -o ./build/lcov/lcov.info.cleaned
Running this command will generate the final coverage file and remove all libraries that do not directly belong to the project.
Putting all together and automating with GitHub Actions.
Now when we finally know all the required steps, we can write a simple script that also can be automated:
# Add build flags sed -i '' '/build_flags\s*=/p; s/build_flags\s*=/-lgcov --coverage/' platformio.ini # Modify memory segments sed -i "s/len\s=\s0x2c200\s-\s0xdb5c/len = 289888/" ~/.platformio/packages/framework-arduinoespressif32/tools/sdk/esp32/ld/memory.ld # Build binaries pio run --environment default # Collect initial coverage lcov -i -d ./.pio/build/default/ -c -o ./build/lcov/lcov.info.initial -gcov-tool ~/.platformio/packages/toolchain-xtensa-esp32/bin/xtensa-esp32-elf-gcov # Running tests pio test -e native # Collect test coverage lcov -d ./.pio/build/native/ -c -o ./build/lcov/lcov.info.test # Merge coverage lcov -o ./build/lcov/lcov.info -a ./build/lcov/lcov.info.initial -a ./build/lcov/lcov.info.test # Clean up libraries lcov --remove ./build/lcov/lcov.info '/usr/include/*' '*.platformio/*' '*/.pio/*' '*/tool-unity/*' '*/test/*' '*/MockArduino/*' -o ./build/lcov/lcov.info.cleaned # Generate HTML report genhtml -p $PWD -o ./build/coverage/ --demangle-cpp ./build/lcov/lcov.info.cleaned
As a bonus step, with the last command, we also generate a fancy HTML web report with easily-understandable coverage information:
Automating with GitHub Actions
My pipeline is actually quite complex and consists of multiple environments with different configurations, and I used matrixes to compile and test multiple targets simultaneously. You can find the final result in the repository:
But in the end, it produces a comment to Pull Request with the final coverage:
In conclusion, automated unit testing and code coverage reports are essential for maintaining the quality of a project. With the help of GitHub Actions, we can automate these processes and ensure that our code is thoroughly tested and properly covered. By following the steps outlined in this article, developers can easily incorporate these practices into their projects and benefit from the resulting improvements in code quality.