🔌 Integration Testing

Integration tests of EmBi modules on target (boards) with Unity running on the MCU. Module under test runs with real hardware and real dependent modules (eg. Gpio + Rcc), no mocks and no register emulation.

Every test case runs after system reset of the MCU - one firmware per test file, flashed once, the firmware executes one test case per boot and resets the MCU before the next one.
Unity output is stored into a result mailbox in RAM (section .noinit, kept over reset), the host reads it by debug probe (SWD). Results do not depend on any peripheral (USART, VCP pins, clock configuration), so a defect of one module cannot hide results of the others.
The framework is independent of the target (MCU, BSP, MCAL, CPU architecture) - it only builds the test firmware, runs the test cases and evaluates the results. Every test set is independent: it implements the target interface (ItCore/IntegrationTesting_Target.h) and initializes everything it needs itself (entry point, clocks, debug freeze, fault handlers, system reset).
Faults reported by the fault handlers of the test set and blocked test cases are reported as failure of the test case (with registers chosen by the test set, eg. PC, LR, CFSR, HFSR) and the execution continues.
The host uses only probe-rs (artifact probe-rs) - any probe supported by probe-rs, no vendor tools. Firmware is built for the MCU of the preset, the host checks the MCU of the board by its ID and flash size registers and selects the probe of the matching board - more boards can be connected at the same time.
Connected boards are detected - the MCU of every probe is identified by the MCU identification of the Ral presets (Bsp/Ral/RalPresets.json). IntegrationTestBoards.json of the project is optional, it only overrides the detected boards (name, preset, probes, parameters).

⚙️ How it works

Part	Description
`IntegrationTesting.cmake`	Included by `Build.cmake` for `CMAKE_BUILD_TYPE=IntegrationTest`. Builds `unity` and `ItCore` for MCU, provides `IntegrationTesting_AddPath()`, `IntegrationTesting_Generate()` and `IntegrationTesting_Add_Test()`. Applies parameters of the board from the boards file and generates the host configuration `IntegrationTesting_Host.cmake` (build folder).
`ItCore/IntegrationTesting.c`	`IntegrationTesting_Run()` called by the entry point of the test set - one test case per boot, system reset between test cases, fault report, result mailbox `integrationTesting_Mailbox`. Target independent (host gcc compiles it).
`ItCore/IntegrationTesting_Target.h`	Target interface implemented by every test set: `ItTarget_Init()`, `ItTarget_SystemReset()`.
`ItCore/unity_config.h`	Unity output (`UNITY_OUTPUT_CHAR`) to the mailbox.
`ItCore/IntegrationTesting_Runner.c.in`	Runner template - table of test cases generated by CMake (no Ruby needed).
`IntegrationTesting_Run.cmake`	Host script: probe selection, MCU check, download, start of test session, reset, waiting for the end (abort of blocked test case), reading of the output (`probe-rs`).
`IntegrationTesting_Check.cmake`	Host script: result of one test function from the stored output.
`IntegrationTesting_Probe.cmake`	Host helper: connected probes (`probe-rs list`), MCU identification of the Ral presets, register reads, probes of the board, probe-rs lookup.
`IntegrationTesting_Detect.cmake`	Host script: detection of connected boards -> `Build/IntegrationTestBoards.json`.
`IntegrationTesting_AllBoards.cmake`	Host script: detection, build and execution of the tests on all connected boards.

Build flow (root CMakeLists.txt -> Build.cmake, CMAKE_BUILD_TYPE=IntegrationTest):

Preset IntegrationTest (CMake/Presets/PlatformPresets.json) replaces the artifacts list of the project ArtifactsConfig.txt by ARM toolchain, unity and probe-rs (ARTIFACTS_HANDLER_REQ_LIST).
MCU build as Debug: flags, Bsp.cmake, Middlewares.cmake, App.cmake. Bsp/Hal/BspMain (application entry point) is skipped by Bsp/Hal/CMakeLists.txt (INTEGRATION_TESTING_AVAILABLE) - the entry point of the test firmware is provided by the test set. Other HAL modules are processed and can register own integration tests.

Every module registers its tests in its own CMakeLists.txt:

#===================== Integration tests configuration ========================#
if(INTEGRATION_TESTING_AVAILABLE STREQUAL "ON")
    IntegrationTesting_AddPath("${CMAKE_CURRENT_SOURCE_DIR}/Tests/IntegrationTests")
endif()

IntegrationTesting_Generate() (instead of project firmware) excludes all module libraries from all target and adds every registered Tests/IntegrationTests/CMakeLists.txt.
For each test: runner is generated, firmware IT_<Name>.elf (+ .hex, .map) is linked from test + SOURCES (ItTarget) + runner + LINK_LIBS (StartUp, module under test, ...) + ItCore/Unity. The framework itself links no MCU library.
CTest tests:
- <Name>.Run - executes the firmware on the board (fixture, RUN_SERIAL). Fails only if the tests were not completely executed (board not found, other MCU, download error, timeout, output overflow, mailbox not kept over reset).
- <Name>.<function> - one per void It_...( void ) function, evaluated from the output (PASS passed, FAIL failed, IGNORE skipped).

Execution on target (<Name>.Run):

host: probe-rs list                            -> connected probes (candidates INTEGRATION_TEST_PROBE)
      probe-rs read b32 <ID address> 1         -> probe whose MCU ID & mask == INTEGRATION_TEST_ID_VALUE
      probe-rs read b16 <flash size address> 1    and flash size == INTEGRATION_TEST_FLASH_SIZE_KB
      probe-rs download --verify IT_<Name>.elf
      probe-rs write b32 <mailbox> ITST, reset  (ITST = start of new test session)

MCU:  startup code -> entry point of the test set (eg. BspMain) -> IntegrationTesting_Run()
      ItTarget_Init (test set) -> mailbox open -> evaluation of the test case interrupted by reset
        -> test case [TestIndex] (Unity) -> TestIndex++ -> ItTarget_SystemReset (test set)
        -> ... -> all test cases done -> summary -> State DONE -> busy loop (no WFI)

host: poll header (probe-rs read b8 <mailbox> 48) until DONE, read Output[Length] -> IT_<Name>.log
      no progress of the test case for CASE_TIMEOUT -> write b32 <Command> ABRT, reset (abort)

Test case interrupted by reset is evaluated after the reset:

Cause	Result
Fault (`IntegrationTesting_ReportFault()` from the fault handler of the test set)	`FAIL: HardFault PC=0x... LR=0x... CFSR=0x... HFSR=0x...` (registers chosen by the test set), next test case
Abort by host (blocked test case)	`FAIL: Timeout - test case blocked, aborted by host (stage N)`, next test case
Reset expected by the test case	the same test case again with stage + 1 (see below)
Other reset (watchdog, brown-out, ...)	`FAIL: Unexpected reset (stage N)`, next test case

🚀 Usage

MCU identification

The framework knows no MCU, board or probe. How an MCU reports itself over the debug port is part of Ral - every MCU preset of Bsp/Ral/RalPresets.json (generated by ral-presets.sh of Bsp_Linker) carries it in its vendor data:

"vendor": {
    "embedbits.com/EmBi": {
        "target": "mcu",
        "identification": {
            "device": "STM32F407VG",
            "core": "Cortex-M4",
            "idAddress": "0xE0042000",
            "idMask": "0x00000FFF",
            "idValue": "0x413",
            "flashSizeAddress": "0x1FFF7A22",
            "flashSizeKB": 1024
        }
    }
}

core - generic probe-rs target used to read the registers before the chip of the board is known.
idAddress / idMask / idValue - ID register of the MCU (STM32: DEV_ID of DBGMCU_IDCODE).
flashSizeAddress / flashSizeKB - 16-bit flash size register [KB], distinguishes variants (xE / xG).
device - default probe-rs chip of the MCU (INTEGRATION_TEST_PROBE_RS_CHIP).

The configuration of an integration test preset takes the ID / flash size check and the probe-rs chip of TARGET_MCU from it (parameters which are not set).

Connected boards

IntegrationTesting_Detect.cmake (Setup menu "Tests" -> "Integration tests" runs it before the boards are listed) reads the ID and flash size registers of every connected probe and selects the MCUs with integration test preset <MCU>_IntegrationTest in the project CMakePresets.json. The result is the boards file of connected boards Build/IntegrationTestBoards.json (not versioned):

cmake -P EmBi_Platform/CMake/IntegrationTesting/IntegrationTesting_Detect.cmake

Connected boards:
  STM32F4DISCOVERY (STM32F407xG_IntegrationTest): 0483:374b:066B... (ID 0x413, 1024 KB), 0483:374b:0674... (ID 0x413, 1024 KB)
Probes without identified MCU (not used):
  0483:374e:0017... (ID not read)

A board without override is named by its MCU (eg. STM32F407xG). Some MCUs share the ID and flash size (STM32F405 / F407 / F415 / F417 xG) - the first one is used and a warning lists the others.

Boards of the project (optional override)

IntegrationTestBoards.json in the project root (next to CMakePresets.json) names the boards, selects their presets and parameters:

{
    "boards": {
        "STM32F4DISCOVERY": {
            "preset": "STM32F407xG_IntegrationTest"
        },
        "NUCLEO_H503RB": {
            "preset": "STM32H503xB_IntegrationTest",
            "probes": [ "<serial number of the probe>" ],
            "cacheVariables": {
                "INTEGRATION_TEST_CASE_TIMEOUT": "20"
            }
        }
    }
}

Board with probes (probe-rs selectors VID:PID[:SN] or serial numbers) gets these probes when they are connected - also probes without identified MCU (the MCU is checked before download).
Board without probes gets all connected probes whose MCU matches its preset - eg. the name of the board (IT_BOARD_<name> selects pins in tests) or the MCU of an ambiguous ID.
cacheVariables are used for parameters which are not set on the command line, in the preset or in the cache.

cmake -P EmBi_Platform/CMake/IntegrationTesting/IntegrationTesting_Detect.cmake
cmake --preset STM32F407xG_IntegrationTest -DINTEGRATION_TEST_BOARD=STM32F4DISCOVERY
cmake --build --preset STM32F407xG_IntegrationTest
ctest --preset STM32F407xG_IntegrationTest    # JUnit: Build/<preset>/IntegrationTestResults.xml

All connected boards (from the project root) - boards are detected, every board is configured with its preset (INTEGRATION_TEST_BOARD=<board>), the firmware is built and executed on every probe of the board:

cmake -P EmBi_Platform/CMake/IntegrationTesting/IntegrationTesting_AllBoards.cmake
cmake -DIT_LABEL=Gpio -P EmBi_Platform/CMake/IntegrationTesting/IntegrationTesting_AllBoards.cmake
# JUnit per probe: Build/<preset>/IntegrationTestResults_<sn>.xml

Without unity artifact (not published yet) - artifacts list without unity:

cmake -S . -B build/it-h503rb -G Ninja -DCMAKE_BUILD_TYPE=IntegrationTest \
      -DARTIFACTS_HANDLER_REQ_LIST="ninja;1.12.0;latest,gcc-arm-none-eabi;latest;latest,probe-rs;latest;latest" \
      -DTARGET_MCU=STM32H503RBTx -DINTEGRATION_TEST_BOARD=NUCLEO_H503RB -DUNITY_ROOT=<Unity>
cmake --build build/it-h503rb && ctest --test-dir build/it-h503rb

Probe selection (<Name>.Run, probe-rs list):

Candidates: environment variable INTEGRATION_TEST_PROBE_SN (more equal boards), otherwise INTEGRATION_TEST_PROBE (probes of the board in the boards file by default), otherwise all connected probes. Probe is given by probe-rs selector (VID:PID[:SN]) or by serial number.
With INTEGRATION_TEST_ID_ADDRESS the ID register (and flash size register) of every candidate is read - the first probe with matching MCU is used (the MCU of the board is always checked before download).
Without MCU ID the first candidate is used - only one probe may be connected, if no candidate is given.

Parameters:

Parameter	Default	Description
`INTEGRATION_TEST_BOARD`	-	Board name, compile definition `IT_BOARD_<name>` selects pins in tests, parameters of the board from the boards file.
`INTEGRATION_TEST_BOARDS_FILE`	`Build/IntegrationTestBoards.json`	Boards file - connected boards (detection).
`INTEGRATION_TEST_MCU_PRESETS_FILE`	`Bsp/Ral/RalPresets.json`	Ral presets with MCU identification.
`INTEGRATION_TEST_PROBE`	probes of the board / all probes	Probes of the board - probe-rs selectors `VID:PID[:SN]` or serial numbers (list).
`INTEGRATION_TEST_PROBE_SN`	-	Environment variable at test time - probe (selector or serial number).
`INTEGRATION_TEST_PROBE_RS_CHIP`	identification `device` / `TARGET_MCU`	probe-rs chip - the first one probe-rs knows (`probe-rs chip info`), otherwise the board defines it (boards file). Names of MCUs are not interpreted.
`INTEGRATION_TEST_ID_ADDRESS`	identification	Address of the MCU ID register, empty = MCU is not checked.
`INTEGRATION_TEST_ID_MASK`	identification / `0xFFFFFFFF`	Mask of the ID bits.
`INTEGRATION_TEST_ID_VALUE`	identification	Expected ID (after mask).
`INTEGRATION_TEST_FLASH_SIZE_ADDRESS`	identification	Address of the 16-bit flash size register [KB], empty = flash size is not checked.
`INTEGRATION_TEST_FLASH_SIZE_KB`	identification	Expected flash size [KB].
`PROBE_RS_EXECUTABLE`	artifact	probe-rs (artifact `probe-rs`, PATH).
`INTEGRATION_TEST_OUTPUT_SIZE`	`4096`	Unity output buffer [B] (RAM of the smallest MCU - H503 has 32 KB).
`INTEGRATION_TEST_TIMEOUT`	`60`	Default timeout of all test cases of the firmware [s], `TIMEOUT` of the test overrides it.
`INTEGRATION_TEST_CASE_TIMEOUT`	`10`	Default timeout of one test case [s], `CASE_TIMEOUT` of the test overrides it.
`INTEGRATION_TEST_RUN_KNOWN_DEFECTS`	`OFF`	Execute tests marked by `IT_KNOWN_DEFECT()`.

Useful:

ctest -L Gpio - tests of one module. A single function cannot be executed alone - the whole firmware runs once (<Name>.Run is added automatically as fixture).
IT_<Name>.elf can be debugged in any debugger as any firmware (-O0 -g3). Without host the firmware starts local session (all test cases, resets between them); hardware breakpoints are kept over system reset. The output is visible in integrationTesting_Mailbox.Output.

Requirements on the target

The framework needs from the project (linker script, startup code, MCU configuration):

Requirement	Why	Failure
Output section `.noinit` in the linker script (`NOLOAD`, in RAM, not zeroed or copied by the startup code)	Result mailbox `integrationTesting_Mailbox` is placed by `__attribute__((section(".noinit")))` - it keeps the session state and the output over system resets between test cases.	Without the section the linker places the mailbox into another section or the startup code clears it: `<Name>.Run` fails with "mailbox is not kept over system reset" (session `LOCL`).
RAM of `.noinit` is not erased by system reset	Hardware erase of RAM on reset (eg. option bytes / reset configuration of the MCU) clears the mailbox the same way.	Same as above.
RAM readable by the debug probe while the core runs	Host polls the mailbox without stopping the test (probe-rs).	Timeout of `<Name>.Run`.
GCC compatible toolchain	`__attribute__((section))`, `-Wl,-Map`, `objcopy -O ihex`, `size`.	Build error.

Example (GNU ld, .noinit after .bss):

  .noinit (NOLOAD) :
  {
    . = ALIGN(4);
    *(.noinit)
    *(.noinit*)
    . = ALIGN(4);
  } >RAM

✍️ Writing tests

Test sets are created from the template by Setup scripts (Setup.sh / Setup.bat: new module, or "Add tests to existing module" - HelperTools/Test_Handler). Module layout:

Bsp/Mcal/Gpio/
├── CMakeLists.txt              # + IntegrationTesting_AddPath(.../Tests/IntegrationTests)
└── Tests/
    ├── UnitTests/              # host tests (UnitTesting)
    └── IntegrationTests/       # this document
        ├── CMakeLists.txt
        ├── ItTest_Gpio.c       # test cases
        ├── ItTarget_Gpio.c     # target interface of the test set
        └── BspMain.h           # entry point called by StartUp

Tests/IntegrationTests/CMakeLists.txt:

# StartUp calls BspMain() - the entry point of the test firmware is provided by the test set
if(NOT TARGET BspMain_Lib)
    add_library(BspMain_Lib INTERFACE)
    target_include_directories(BspMain_Lib INTERFACE ${CMAKE_CURRENT_SOURCE_DIR})
endif()

IntegrationTesting_Add_Test(
    NAME            Gpio
    TEST_SOURCE     ItTest_Gpio.c
    SOURCES         ItTarget_Gpio.c     # target interface of the test set
    LINK_LIBS       StartUp_Lib         # startup code (calls BspMain)
                    Gpio_Lib            # module under test (+ its dependencies)
    TIMEOUT         30                  # all test cases [s]
    CASE_TIMEOUT    5                   # one test case [s]
)

Target interface (ItTarget_Gpio.c) - the framework calls only these two functions, everything else (including the entry point) is owned by the test set:

Function	Implemented by	Description
entry point (eg. `BspMain()`)	test set	Called by the startup code, calls `IntegrationTesting_Run()` (never returns).
`ItTarget_Init()`	test set	Every boot, before the test case: fault handlers, debug freeze of the peripherals used by the tests (probe-rs halts the core on every mailbox access), clocks, ...
`ItTarget_SystemReset()`	test set	Reset of the target between the test cases (shall not return).
fault handlers	test set	Collect registers and call `IntegrationTesting_ReportFault( name, regs, cnt )` (never returns).

The template implements the interface for Cortex-M / STM32 (Nvic API, SCB fault status, SecureFault only on ARMv8-M main). Another architecture needs only another ItTarget_<Module>.c, the framework stays unchanged.

Test file:

#include "unity.h"
#include "IntegrationTesting.h"
#include "Gpio_Port.h"

#if defined(IT_BOARD_NUCLEO_H503RB) || defined(IT_BOARD_NUCLEO_H533RE)
    #define IT_GPIO_FREE_PORT   ( GPIO_PORT_A )
    ...
#else
    #error "Board of Gpio integration tests is not defined (INTEGRATION_TEST_BOARD)."
#endif

void setUp( void )    { ... }
void tearDown( void ) { /* return backup domain / option bytes changes */ }

void It_Gpio_Set_PinPull_InputPullUp_PinReadsHigh( void ) { ... }

Test case expecting reset (eg. watchdog) - every expected reset executes the test case again with incremented stage. Reset source flags are target specific - the test case clears them itself:

void It_Iwdg_Timeout_ResetsMcu( void )
{
    if( 0u == IntegrationTesting_Get_Stage() )
    {
        TEST_ASSERT_EQUAL( RCC_REQUEST_OK, Rcc_Set_ResetSourceClear() );
        IntegrationTesting_Set_ResetExpected();
        /* start watchdog, wait for reset */
        TEST_FAIL_MESSAGE( "Reset did not occur" );
    }
    else
    {
        rcc_FlagState_t iwdgReset = RCC_FLAG_INACTIVE;
        TEST_ASSERT_EQUAL( RCC_REQUEST_OK, Rcc_Get_ResetSource( RCC_RESET_SRC_IWDG, &iwdgReset ) );
        TEST_ASSERT_EQUAL( RCC_FLAG_ACTIVE, iwdgReset );
    }
}

Rules:

Test cases (ItTest_<Module>.c) test only through public API of the modules (<Mod>_Port.h) - no LL, no direct register access (MCAL layering). Dependencies are real modules, never mocks. Target specific code of the test set (debug freeze, fault registers) belongs to ItTarget_<Module>.c.
Every test case starts after system reset (core, peripherals, NVIC and clocks in reset state), so the order of tests does not matter. System reset does not reset the backup domain (RTC, LSE, TAMP backup registers) and option bytes - a test changing them returns them in tearDown().
Board dependent resources (pins, LEDs, loopback wiring) are selected by IT_BOARD_<name>, unknown board ends by #error. Tests needing external wiring document it in the file header.
Test names as in unit tests: test_<Function>_<Condition>_<ExpectedResult>.
Keep output small (INTEGRATION_TEST_OUTPUT_SIZE).

🔭 Next steps

Integration tests of other modules (Rcc, Exti, Nvic, Tim, Usart, Gpdma, Adc, I2c, Iwdg).
Artifact unity (see UnitTesting/README.md).
Renode: the same IT_<Name>.elf, mailbox read by Renode monitor (sysbus ReadDoubleWord).

⚙️ How it works​

🚀 Usage​

MCU identification​

Connected boards​

Boards of the project (optional override)​

Requirements on the target​

✍️ Writing tests​

🔭 Next steps​