Skip to main content

πŸ”Œ Integration Testing

Integration tests of EmBi modules on target (boards) with Unity running on the MCU. Module under test runs with real hardware and real dependent modules (eg. Gpio + Rcc), no mocks and no register emulation.

  • Every test case runs after system reset of the MCU - one firmware per test file, flashed once, the firmware executes one test case per boot and resets the MCU before the next one.
  • Unity output is stored into a result mailbox in RAM (section .noinit, kept over reset), the host reads it by debug probe (SWD). Results do not depend on any peripheral (USART, VCP pins, clock configuration), so a defect of one module cannot hide results of the others.
  • The framework is independent of the target (MCU, BSP, MCAL, CPU architecture) - it only builds the test firmware, runs the test cases and evaluates the results. Every test set is independent: it implements the target interface (ItCore/IntegrationTesting_Target.h) and initializes everything it needs itself (entry point, clocks, debug freeze, fault handlers, system reset).
  • Faults reported by the fault handlers of the test set and blocked test cases are reported as failure of the test case (with registers chosen by the test set, eg. PC, LR, CFSR, HFSR) and the execution continues.
  • The host uses only probe-rs (artifact probe-rs) - any probe supported by probe-rs, no vendor tools. Firmware is built for the MCU of the preset, the host checks the MCU of the board by its ID and flash size registers and selects the probe of the matching board - more boards can be connected at the same time.
  • Connected boards are detected - the MCU of every probe is identified by the MCU identification of the Ral presets (Bsp/Ral/RalPresets.json). ItBoards.json of the project is optional, it only overrides the detected boards (name, preset, probes, parameters).

βš™οΈ How it works​

PartDescription
IntegrationTesting.cmakeIncluded by Build.cmake for CMAKE_BUILD_TYPE=IntegrationTest. Builds unity and ItCore for MCU, provides IntegrationTesting_AddPath(), IntegrationTesting_Generate() and IntegrationTesting_Add_Test(). Applies parameters of the board from the boards file and generates the host configuration IntegrationTesting_Host.cmake (build folder).
ItCore/IntegrationTesting.cIntegrationTesting_Run() called by the entry point of the test set - one test case per boot, system reset between test cases, fault report, result mailbox integrationTesting_Mailbox. Target independent (host gcc compiles it).
ItCore/IntegrationTesting_Target.hTarget interface implemented by every test set: ItTarget_Init(), ItTarget_SystemReset().
ItCore/unity_config.hUnity output (UNITY_OUTPUT_CHAR) to the mailbox.
ItCore/IntegrationTesting_Runner.c.inRunner template - table of test cases generated by CMake (no Ruby needed).
IntegrationTesting_Run.cmakeHost script: probe selection, MCU check, download, start of test session, reset, waiting for the end (abort of blocked test case), reading of the output (probe-rs).
IntegrationTesting_Check.cmakeHost script: result of one test function from the stored output.
IntegrationTesting_Probe.cmakeHost helper: connected probes (probe-rs list), MCU identification of the Ral presets, register reads, probes of the board, probe-rs lookup (the artifact in the version configured for the integration test preset, downloaded in that version if it is missing; never probe-rs of the system or the newest cached version).
IntegrationTesting_Lock.cmakeHost helper: probe lock - one client of a probe at a time (debugging, tests, detection, pipeline jobs), see Probe lock.
IntegrationTesting_Remote.cmakeHost helper: IT agents of ItHosts.json and the IT agent interface - embi-agent of the agent started over SSH, see Boards of IT agents.
IntegrationTesting_DebugServer.cmakeHost script: debug server of a probe of this machine (probe-rs dap-server under the probe lock) - a board of an IT agent is served by embi-agent debug-server.
IntegrationTesting_Package.cmakeHost script: the test package of a build (input of run-it of an IT agent and of the pipeline jobs).
IntegrationTesting_Detect.cmakeHost script: detection of connected boards -> Build/IntegrationTestBoards.json.
IntegrationTesting_AllBoards.cmakeHost script: detection, build and execution of the tests on all connected boards.

Build flow (root CMakeLists.txt -> Build.cmake, CMAKE_BUILD_TYPE=IntegrationTest):

  1. Preset IntegrationTest (CMake/Presets/PlatformPresets.json) replaces the artifacts list of the project ArtifactsConfig.txt by ARM toolchain, unity and probe-rs (ARTIFACTS_HANDLER_REQ_LIST).
  2. MCU build as Debug: flags, Bsp.cmake, Middlewares.cmake, App.cmake. Bsp/Hal/BspMain (application entry point) is skipped by Bsp/Hal/CMakeLists.txt (INTEGRATION_TESTING_AVAILABLE) - the entry point of the test firmware is provided by the test set. Other HAL modules are processed and can register own integration tests.
  3. Every module registers its tests in its own CMakeLists.txt:
    #===================== Integration tests configuration ========================#
    if(INTEGRATION_TESTING_AVAILABLE STREQUAL "ON")
    IntegrationTesting_AddPath("${CMAKE_CURRENT_SOURCE_DIR}/Tests/IntegrationTests")
    endif()
  4. IntegrationTesting_Generate() (instead of project firmware) excludes all module libraries from all target and adds every registered Tests/IntegrationTests/CMakeLists.txt.
  5. For each test: runner is generated, firmware IT_<Name>.elf (+ .hex, .map) is linked from test + SOURCES (ItTarget) + runner + LINK_LIBS (StartUp, module under test, ...) + ItCore/Unity. The framework itself links no MCU library.
  6. CTest tests:
    • <Name>.Run - executes the firmware on the board (fixture, RUN_SERIAL). Fails only if the tests were not completely executed (board not found, other MCU, download error, timeout, output overflow, mailbox not kept over reset).
    • <Name>.<function> - one per void It_...( void ) function, evaluated from the output (PASS passed, FAIL failed, IGNORE skipped).

Execution on target (<Name>.Run):

host: probe-rs list -> connected probes (candidates INTEGRATION_TEST_PROBE)
probe-rs read b32 <ID address> 1 -> probe whose MCU ID & mask == INTEGRATION_TEST_ID_VALUE
probe-rs read b16 <flash size address> 1 and flash size == INTEGRATION_TEST_FLASH_SIZE_KB
probe-rs download --verify IT_<Name>.elf
probe-rs write b32 <mailbox> ITST, reset (ITST = start of new test session)

MCU: startup code -> entry point of the test set (eg. BspMain) -> IntegrationTesting_Run()
ItTarget_Init (test set) -> mailbox open -> evaluation of the test case interrupted by reset
-> Progress = 0x0A000000 | TestIndex -> wait for the attach of the probe trace (session HTRC)
-> test case [TestIndex] (Unity) -> TestIndex++
-> Progress |= 0x00800000 (case done) -> wait for Command NEXT and the end of its probe session
(session HTRC) -> ItTarget_SystemReset (test set)
-> ... -> all test cases done -> summary -> State DONE -> busy loop (no WFI)
session start / abort (session HTRC, phase STLE): wait for Command NEXT and the end of its probe
session -> ItTarget_SystemReset, no test case

host: write b32 <mailbox> ITST TRCE 0 (start of a session polled by probe trace, Length 0 written after State:
on a RAM with ECC the last write before a reset is lost), reset, read header once,
write b32 <Session> HTRC NEXT
probe-rs trace <Progress> in the background - one probe session per test case, the host kills it at
"case done", writes b32 <Session> HTRC NEXT and starts the next one; no progress for CASE_TIMEOUT -> abort
until TestIndex == TestCount, then read header until DONE, read Output[Length] -> IT_<Name>.log
abort: write b32 <Session> HTRC ABRT, reset, write b32 <Session> HTRC NEXT

Every probe-rs call (read, write, reset) halts the core on its attach and on its end (probe-rs clears the hardware breakpoints, about 30 ms each). A halt in the middle of a test case loses the data of a running transfer (a UART byte by overrun, for example), so the host does not poll the mailbox by probe-rs read during the test cases (AB#2247): probe-rs trace keeps one probe session and reads the Progress word (after Output) without halting the core. Its attach still halts the core - the firmware of a session started with TRCE (session HTRC) clears the sticky halt flag and starts a test case after the halt of the attach of the probe trace (handler per core, ItCore/IntegrationTesting_Core.h: Cortex-M4 / M7 by DFSR.HALTED; Cortex-M33 by DHCSR.S_RESTART_ST, its DFSR.HALTED stays cleared). After the test case it sets the flag "case done" in Progress: the host kills the probe trace (the end of a probe session halts the core, a killed probe process does not end it; probe-rs trace does not end on an interrupt) and writes the command NEXT, the firmware resets the MCU after the halt of the end of that probe session (AB#2331: no probe session ends during the next test case - the probe trace survives the reset on Cortex-M33 / M4, Cortex-M7 resets end it late). The mailbox is written to the RAM completely before the flag "case done": a failed open of the probe resets the MCU (pin reset), on a RAM with ECC (STM32H7 AXI SRAM) the last write before a reset is lost otherwise. The session start and an abort use several probe sessions in a row: the host writes NEXT after them, the firmware resets the MCU without a test case (phase STLE). Every wait of the firmware ends after several seconds at most. A test case continued after its expected reset waits for a halt as every test case - the probe trace may survive that reset (the wait ends by its timeout). The line feed in the most significant byte of Progress makes probe-rs write every record to its line buffered output at once. Firmware without the Progress word is polled by probe-rs read (after 5 s without a record); environment variable INTEGRATION_TEST_POLL=read selects that polling always.

Test case interrupted by reset is evaluated after the reset:

CauseResult
Fault (IntegrationTesting_ReportFault() from the fault handler of the test set)FAIL: HardFault PC=0x... LR=0x... CFSR=0x... HFSR=0x... (registers chosen by the test set), next test case
Abort by host (blocked test case)FAIL: Timeout - test case blocked, aborted by host (stage N), next test case
Reset expected by the test casethe same test case again with stage + 1 (see below)
Other reset (watchdog, brown-out, ...)FAIL: Unexpected reset (stage N), next test case

πŸš€ Usage​

MCU identification​

The framework knows no MCU, board or probe. How an MCU reports itself over the debug port is part of Ral - every MCU preset of Bsp/Ral/RalPresets.json (generated by ral-presets.sh of Bsp_Linker) carries it in its vendor data:

"vendor": {
"embedbits.com/EmBi": {
"target": "mcu",
"identification": {
"device": "STM32F407VG",
"core": "Cortex-M4",
"idAddress": "0xE0042000",
"idMask": "0x00000FFF",
"idValue": "0x413",
"flashSizeAddress": "0x1FFF7A22",
"flashSizeKB": 1024
}
}
}
  • core - generic probe-rs target used to read the registers before the chip of the board is known.
  • idAddress / idMask / idValue - ID register of the MCU (STM32: DEV_ID of DBGMCU_IDCODE).
  • flashSizeAddress / flashSizeKB - 16-bit flash size register [KB], distinguishes variants (xE / xG).
  • device - default probe-rs chip of the MCU (INTEGRATION_TEST_PROBE_RS_CHIP).

The configuration of an integration test preset takes the ID / flash size check and the probe-rs chip of TARGET_MCU from it (parameters which are not set).

Connected boards​

IntegrationTesting_Detect.cmake (Setup menu "Tests" -> "Integration tests" runs it before the boards are listed) reads the ID and flash size registers of every connected probe and selects the MCUs with integration test preset <MCU>_IntegrationTest in the project CMakePresets.json. The result is the boards file of connected boards Build/IntegrationTestBoards.json (not versioned):

cmake -P EmBi_Platform/CMake/IntegrationTesting/IntegrationTesting_Detect.cmake
Connected boards:
STM32F405xG (STM32F405xG_IntegrationTest): 0483:374b:066B... (ID 0x413, 1024 KB), 0483:374b:0674... (ID 0x413, 1024 KB)
Probes without identified MCU (not used):
0483:374e:0017... (ID not read)

A board without override is named by its MCU (eg. STM32F407xG). Some MCUs share the ID and flash size (STM32F405 / F407 / F415 / F417 xG, STM32G411xB / G431xB / G441xB) - the first one whose device probe-rs knows is used (the firmware can be executed, eg. STM32G431xB instead of STM32G411xB), otherwise the first one; a warning lists the others.

Note: boards are always named by their MCU - never by the development board (NUCLEO_*, *DISCOVERY). Test sets of modules select pins and peripherals of the board by IT_BOARD_<MCU> (INTEGRATION_TEST_BOARD) and list all MCUs with the same ID and flash size in one block (eg. IT_BOARD_STM32G471xE ... IT_BOARD_STM32G484xE for NUCLEO-G474RE), so the board works without ItBoards.json. A test set without a block for the MCU stops with #error Board of ... integration tests is not defined - the block is added to the test set.

Besides the boards (configuration of the firmware: preset, parameters) the boards file lists the targets - one physical board per probe, of this machine ("host": "local") and of every IT agent of ItHosts.json (see Boards of IT agents). A target has the ID <board>@<host>:<probe> and a name of the board behind the probe (label): the fixture of the probe (Board fixtures), on an IT agent its boards.conf, otherwise the end of the serial number. A run of one target gives its probe (INTEGRATION_TEST_PROBE), so it uses exactly one physical board. A probe used by another client (Probe lock) is not read - its target of the previous detection stays listed with usedBy. An IT agent whose detection failed (not reachable, agent tools not installed) is listed in failed (host, SSH target, error) - the board inputs of VS Code show it when they list no board.

Two boards of one MCU are two targets of one board. When the probes of such a board are not defined in ItBoards.json, the board is marked "ambiguous": the boards may have a different wiring, so INTEGRATION_TEST_PROBE does not default to its probes and a run without a probe stops when more probes match (the first one is never taken). Boards with a different wiring or other attached ICs get two board names and their fixtures - see Board fixtures.

Board fixtures​

The fixture of a physical board is what is connected to it - the wiring (eg. SPI1 to SPI2) and the attached ICs (eg. an I2C EEPROM). It is a hardware change, so it is configured manually per serial number of the probe (unique per board - the ST-LINK of a Nucleo / Discovery is a part of the board; a probe without a serial number, the ST-LINK/V2 of the first STM32F4DISCOVERY, is configured by its board name in boards):

The board configuration of the project is ItBoards.json in the project root (it was IntegrationTestBoards.json, which is not read any more - the detection warns when only the old file exists):

{
"probes": {
"001700084D4B500B20373831": { "host": "TestAgent", "name": "NUCLEO-H503RB #1", "board": "STM32H503xB",
"features": { "SPI1_SPI2_LOOP": {} } },
"0026003F3133510237363734": { "host": "local", "name": "NUCLEO-H503RB #2", "board": "STM32H503xB_Eeprom",
"features": { "EEPROM_24LC256": { "I2C": "1", "ADDRESS": "0x50" } } }
},
"boards": {
"STM32F407xG": { "host": "local", "name": "STM32F4DISCOVERY", "features": { "I2C1_I2C2_LOOP": {} } }
}
}
MemberDescription
hostlocal (a board of this machine, the default) or the IT agent of the board (name of ItHosts.json) - debugging uses the probe-rs of this machine or the debug server of the agent.
nameName of the physical board in the board lists (VS Code, Setup) and messages.
boardBoard name of the firmware - the MCU (default) or a variant (eg. STM32H503xB_Eeprom): the probe is the board of that name with the preset of its MCU (identified on the probe, or preset of the board in boards).
featuresWiring and attached ICs: <FEATURE> (letters, digits, _) with parameters <KEY>: <value> (eg. the I2C peripheral and the address of an IC). Probes of one board have the same features - one firmware is built per board.
definesOwn defines of the board for its builds, given as written: <NAME>: <value> ("" = <NAME> without a value). One board name has one set of defines.
staletrue: the board is no longer in the configuration of its IT agent - the entry is kept, but not used (detection, debugging).

Where the boards are configured:

  • Boards of the IT agents - the inventory of the integration test pipeline AzurePipelines/IntegrationTestBoards.yml of EmBi_Tester (versioned, label and features of every board), the only source. The pipeline builds the packages with them and puts the configuration of the boards of every agent on the agent (~/_tools/embi/ItBoards.json, this format without host). Every detection of this machine reads it from the agent (embi-agent detect) and updates the boards of that agent in ItBoards.json of the project: every board of the agent is taken over with "host": "<agent>", a board of the agent that the agent no longer has gets "stale": true; the boards of this machine and of other agents are not touched, the file is written only when it changes (an agent that is not reached changes nothing).
  • Boards of this machine - "host": "local" entries of ItBoards.json, written by hand.

A probe with a serial number that no configuration has is listed by the detection as a new board with its decoded MCU (New boards (serial number not in the board configuration)) - to be configured.

The configuration of a board gives the compile definitions below (IntegrationTesting_BoardConfig.cmake) to the integration test firmware of the board (IntegrationTesting.cmake) and to the Debug / Release build of a picked board (TARGET_BOARD, a build folder of its own Build/<MCU>_<Debug|Release>_<board> - Build_Handler BUILD with BOARD, VS Code EmBi: Build for board, the Debug firmware of EmBi: Debug on board); a build without a board has none of them. Everything of the integration tests has the prefix IT_, the own defines are given as written:

DefinitionMeaning
IT_BOARD_<board>Board name (as without a fixture).
IT_BOARD_<MCU>MCU of a variant (board name other than the MCU) - the test sets keep their blocks of the MCU, only a test set that uses the variant needs a block of it (checked before the block of the MCU).
IT_FIXTUREThe board has a fixture configured.
IT_FEATURE_<FEATURE>The board has the feature.
IT_FEATURE_<FEATURE>_<KEY>=<value>Parameter of the feature (eg. IT_FEATURE_EEPROM_24LC256_ADDRESS=0x50).
<NAME>=<value>Own define of the board (defines).

IT_BOARD_<board> and IT_BOARD_<MCU> are given to the integration test firmware only.

A test that needs a feature is ignored on a board without it (skipped with the reason in the JUnit file). A board without a fixture (IT_FIXTURE not defined) keeps the behavior of the test set before the fixtures - the test set expects its documented wiring:

void It_Spi_Poll_FullDuplex_DataExchanged( void )
{
#if defined(IT_FEATURE_SPI1_SPI2_LOOP) || \
!defined(IT_FIXTURE)
/* SPI1 and SPI2 are connected */
#else
TEST_IGNORE_MESSAGE( "Board without the fixture SPI1_SPI2_LOOP" );
#endif
}

Boards of IT agents​

An IT agent is a machine with debug probes other than this one - eg. a Linux host of the integration test pipeline. Its boards are detected, debugged and tested from this machine over SSH. ItHosts.json in the project root (optional, not versioned - the hosts and accounts of the developer) lists the agents:

{
"hosts": {
"TestAgent": "root@192.168.1.100",
"Lab": { "target": "pi@lab-host:2222", "tools": "/opt/embi/embi-agent" }
}
}
MemberDescription
<name>Name of the host in the board lists (letters, digits, _, -, .; local is this machine).
target<user>@<host>[:<SSH port>] - the key of this machine is authorized there (no password prompt, BatchMode).
toolsembi-agent of the agent (the IT agent interface), default $HOME/_tools/embi/embi-agent or /home/*/_tools/embi/embi-agent of the agent (installed by the pipeline ItAgentUpdate of EmBi_Tester).

The agent needs an SSH server and its agent tools embi-agent (EmBi_Tester, installed with probe-rs, the identification of the MCUs of all families and the board configuration of the agent by the pipeline ItAgentUpdate). This machine only starts them (IT agent interface) - nothing of this machine is copied to the agent and executed there; ssh and scp of the system are used (OpenSSH, Windows 10 and newer has them). An agent that is not reachable, has no agent tools or another interface version is reported and its boards are not listed.

  • Detection (embi-agent detect): the agent reads its probes (probe-rs of the version of this machine) and identifies their MCUs by the identification of all families; this machine forms the boards with the presets of this project - a board of an MCU of another family is listed as not usable here (... - family STM32U5 is not in this project), without a warning also when its fixture gives a board. Names and fixtures come from the board configuration of the agent (the inventory of the pipeline).
  • Debugging (embi-agent debug-server, VS Code EmBi: Debug on board): the debug server of the agent behind ssh -tt -L <port of this machine>:127.0.0.1:<port of the agent>.
  • Serial monitor (embi-agent serial-server, VS Code task EmBi: Serial monitor on board): the virtual COM port of the ST-LINK of the board on 127.0.0.1:<port of this machine> (the same tunnel) - the Serial Monitor of VS Code connects there in TCP mode. The probe is not locked: the board is debugged and tested meanwhile.
  • Flash (embi-agent flash, VS Code task EmBi: Flash): the build of the picked board is copied into a folder of the agent (new-package), flashed there (probe lock and probe-rs of the agent) and the folder is removed (remove-package). The option bytes of the firmware are written after it (embi-agent option-bytes), also by the debug configurations of the board before the debug server.
  • Integration tests on a board of an IT agent are built here and executed there: the preset is configured with INTEGRATION_TEST_HOST=<name of ItHosts.json> (and the probe of the target, INTEGRATION_TEST_PROBE), every <Name>.Run packs its test set (Test package, label <Name>), copies it into a folder of the agent (new-package), runs it there (run-it - probe lock and probe-rs of the agent, holder of this machine), copies the output (and the coverage data) back and removes the folder (remove-package). The test functions, the JUnit file and the coverage report are evaluated here as for a local board. The board lists of Setup (Tests -> Integration tests) give it for a picked board; All connected boards tests the boards of the agents too (environment variables INTEGRATION_TEST_HOST, INTEGRATION_TEST_HOST_TOOLS and INTEGRATION_TEST_PROBE_SN of the run, JUnit file IntegrationTestResults_<host>_<sn>.xml).
cmake --preset STM32H503xB_IntegrationTest -DINTEGRATION_TEST_BOARD=STM32H503xB \
-DINTEGRATION_TEST_PROBE=0483:374e:001700084D4B500B20373831 -DINTEGRATION_TEST_HOST=TestAgent
cmake --build --preset STM32H503xB_IntegrationTest
ctest --preset STM32H503xB_IntegrationTest -L Gpio

IT agent interface​

This machine talks to an IT agent only through embi-agent - the tools of the agent, installed on the agent from EmBi_Tester by a pipeline (~/_tools/embi of the pipeline user, with the identification of the MCUs of all families and the configuration of the boards of the agent). Nothing else of this machine is copied to the agent and executed there; the only data an operation sends are the firmware of a debug session (probe-rs protocol), a test package (run-it, with the host scripts of this platform), a firmware to flash (flash) and a firmware with OptionBytes_Flash.cmake of this platform (option-bytes). The interface is defined here, the agent implements it.

ItemDefinition
VersionIT_AGENT_INTERFACE_VERSION = 1 (IntegrationTesting_Remote.cmake). Every result carries "interface"; this machine refuses another version with an error naming both (update the agent tools or the platform).
Callssh <target> sh -c '<locator>' embi-agent <command> [--<option> <value>]... - <locator> runs tools of the host in ItHosts.json (absolute path of embi-agent on the agent), otherwise the first executable $HOME/_tools/embi/embi-agent or /home/*/_tools/embi/embi-agent; none: exit code 127, embi-agent is not installed on this IT agent.
ResultOne line EMBI-AGENT <JSON> on the standard output (compact JSON on one line, IT_AGENT_RESULT_PREFIX); every other line is a log for people. Exit code 0 = done; otherwise the result has "error": "<text>" when the command got so far. debug-server and serial-server have no result line when they start their server.
CleanupEvery command removes the folders of earlier operations older than 24 hours (/tmp/embi-it-*, /tmp/embi-dap-*) - left by a terminated task or a broken connection.
Probe lockThe agent takes the lock of a probe (Probe lock) for every operation on it; --holder is the holder written for the developer machine (<login>@<computer> (<purpose>)). serial-server is no operation on the probe: it takes the lock of the virtual COM port of the probe only (serial-<key>.lock / .holder in the folder of the probe locks, the same protocol), so the board is debugged and tested while its output is monitored.
CommandOptionsResult / behavior
version-{"interface": 1, "tools": "<commit of EmBi_Tester>", "host": "<host name>"}
detect[--probe-rs <version>] [--holder <text>]{"interface": 1, "host": "<host name>", "probes": [ <probe> ... ], "config": <configuration>} - the connected probes and the board configuration of the agent; probe-rs of the version (default: the newest one of the agent), the holder of the lock while a probe is read.
debug-server--probe <VID:PID:SN> --port <port> --probe-rs <version> --holder <text>probe-rs dap-server of the probe on the port of 127.0.0.1 of the agent for one session (this machine forwards the port, ssh -L), under the probe lock; prints EmBi: debug server listening on port <port> when it listens and ends with the session or the connection. probe-rs of the given version (the version of this machine) is installed on the agent when it is missing.
serial-server--probe <VID:PID:SN> --port <port> --holder <text> [--baud <rate>]The virtual COM port of the probe (/dev/serial/by-id/*_<serial number>-if<interface>, the first one of a probe with two) on the port of 127.0.0.1 of the agent (this machine forwards the port, ssh -L, and the Serial Monitor of VS Code connects to it in TCP mode): raw, 8N1, no flow control, --baud (default 115200). One client at a time, the next one after a disconnect; the output of the board without a client is dropped. One serial monitor per probe - a second one ends with the holder of the first one (the same holder waits for its previous session as debug-server). Prints EmBi: serial server listening on port <port> when it listens and ends with the connection or when the virtual COM port is gone. Added to version 1 (the other commands are unchanged); agent tools without it answer Unknown command.
new-package-{"interface": 1, "dir": "/tmp/embi-it-<random>"} - an empty folder of the agent for a test package (copied there by scp).
run-it--package <dir> --probe <VID:PID:SN> --holder <text> [--label <ctest label regex>] [--lock-wait <s>]Runs the test package (Test package) on the probe (ItRun.cmake of the agent): {"interface": 1, "result": <exit code of ctest>, "notConnected": <true when no probe reads the MCU of the package>, "junit": "<dir>/IntegrationTestResults.xml"}; the logs are in <dir>/logs, the coverage data in <dir>/coverage/dumps.
remove-package--package <dir>Removes the folder of new-package (only /tmp/embi-it-*): {"interface": 1}.
flash--package <dir> --firmware <file> --probe <VID:PID:SN> --chip <probe-rs chip> --probe-rs <version> --holder <text> [--lock-wait <s>]Flashes the firmware <dir>/<file> (copied into the folder of new-package by scp) to the probe (probe-rs download) and resets the MCU, under the probe lock: {"interface": 1}. A probe used by another client is waited for --lock-wait seconds at most (default 30), then the command fails with its holder. probe-rs of the given version is installed on the agent when it is missing. Added to version 1 (the other commands are unchanged); agent tools without it answer Unknown command.
option-bytes--package <dir> --firmware <file> --script <file> --probe <VID:PID:SN> --chip <probe-rs chip> --probe-rs <version> --holder <text> [--lock-wait <s>]Writes the option bytes of the firmware <dir>/<file> (ELF section .option_bytes) to the probe by the script <dir>/<script> (OptionBytes_Flash.cmake of this platform in script mode, copied with the firmware), under the probe lock: only the option bytes which differ from the chip are written, then the MCU is reset; a chip with the values is not written: {"interface": 1}. --lock-wait and probe-rs as flash. Added to version 1 (the other commands are unchanged); agent tools without it answer Unknown command - this machine warns, the option bytes are not written.

<probe> of detect:

{ "probe": "0483:374e:004A00383133510837363734", "serial": "004A00383133510837363734",
"mcus": [ "STM32U5A5xJ" ], "family": "STM32U5", "device": "STM32U5A5AJ",
"id": "0x481", "flashKB": 4096, "text": "ID 0x481, 4096 KB",
"label": "<name of boardNames of the agent>", "usedBy": "" }
  • mcus - the MCUs (Ral preset names) whose identification matches, by the identification of all families; empty when the MCU is not identified (text tells why). More MCUs = an ambiguous ID (eg. STM32F405xG / STM32F407xG).
  • usedBy - the holder of the lock of a probe used by another client: the probe is not read then (mcus empty).
  • label - the name of the board in the board names file of the agent (/opt/hwgui/boards.conf, the panel), "" if none.

<configuration> of detect - the board configuration of the agent, the boards of the inventory of the pipeline (the format of ItBoards.json without host):

{ "probes": { "<serial number>": { "name": "<abstract name>", "board": "<board name>",
"features": { "<X>": { "<KEY>": "<value>" } },
"defines": { "<NAME>": "<value>" } } },
"boards": { "<board name>": { "name": "<abstract name>", "features": {}, "defines": {} } } }

Test package​

The integration tests of a build packed for a machine that builds nothing - the input of run-it of an IT agent and of the jobs of the integration test pipeline (the runner ItRun.cmake of EmBi_Tester). The packer is IntegrationTesting_Package.cmake; the format is a part of the IT agent interface.

<package>/
manifest.json what is where (below)
firmware/IT_<Name>.elf test firmware
firmware/IT_<Name>.nm lines "<address> B integrationTesting_Mailbox" (and "... integrationTesting_Coverage"
of a firmware with the coverage) - the runner has no arm-none-eabi-nm
host/ the host scripts of this platform (Run, Probe, Check and their includes) in their layout
of EmBi_Platform - they always match the platform the firmware was built with
logs/ output of the firmware (written by the runner)
coverage/ only with INTEGRATION_TEST_COVERAGE: notes, sources and coverage scripts, dumps
manifest.jsonContent
schemaVersion of the manifest (1) - the runner refuses another one.
created, board, sourceUTC time of the packing, the board name (IT_BOARD_<name>), module / commit / run and the commit of EmBi_Platform the firmware was built from.
probeRsVersion of probe-rs the runner uses (installs it when missing).
hostConfigValues of IntegrationTesting_Host.cmake of the build (chip, ID and flash size registers, lock timeout) without the values of the machine of the build (IT_PROBE_RS, IT_PROBE, IT_HOST, IT_HOST_TOOLS, the coverage tools) - the runner writes the file again with its probe-rs and probe.
testsEvery CTest test: name, kind (run / check), host script, parameters (paths relative to the package), properties.
coverageOnly with the coverage: the build folder of the firmware and the dumps.
filesEvery file of the package with its SHA-256 (checked by the runner).

Boards of the project (optional override)​

ItBoards.json in the project root (next to CMakePresets.json) is not needed - it only selects another MCU of an ambiguous ID, probes or parameters of the boards. Board names stay MCU names:

{
"boards": {
"STM32F407xG": {
"preset": "STM32F407xG_IntegrationTest"
},
"STM32H503xB": {
"preset": "STM32H503xB_IntegrationTest",
"probes": [ "<serial number of the probe>" ],
"cacheVariables": {
"INTEGRATION_TEST_CASE_TIMEOUT": "20"
}
}
}
}
  • Board with probes (probe-rs selectors VID:PID[:SN] or serial numbers) gets these probes when they are connected - also probes without identified MCU (the MCU is checked before download).
  • Board without probes gets all connected probes whose MCU matches its preset - eg. another MCU of an ambiguous ID (STM32F407xG instead of STM32F405xG); the name of the board selects pins in tests (IT_BOARD_<name>).
  • cacheVariables are used for parameters which are not set on the command line, in the preset or in the cache.
cmake -P EmBi_Platform/CMake/IntegrationTesting/IntegrationTesting_Detect.cmake
cmake --preset STM32F407xG_IntegrationTest -DINTEGRATION_TEST_BOARD=STM32F407xG
cmake --build --preset STM32F407xG_IntegrationTest
ctest --preset STM32F407xG_IntegrationTest # JUnit: Build/<preset>/IntegrationTestResults.xml

All connected boards (from the project root) - boards are detected, every board is configured with its preset (INTEGRATION_TEST_BOARD=<board>), the firmware is built and executed on every probe of the board:

cmake -P EmBi_Platform/CMake/IntegrationTesting/IntegrationTesting_AllBoards.cmake
cmake -DIT_LABEL=Gpio -P EmBi_Platform/CMake/IntegrationTesting/IntegrationTesting_AllBoards.cmake
# JUnit per probe: Build/<preset>/IntegrationTestResults_<sn>.xml (IT agent: IntegrationTestResults_<host>_<sn>.xml)

Without unity artifact (not published yet) - artifacts list without unity:

cmake -S . -B build/it-h503rb -G Ninja -DCMAKE_BUILD_TYPE=IntegrationTest \
-DARTIFACTS_HANDLER_REQ_LIST="ninja;1.12.0;latest,gcc-arm-none-eabi;latest;latest,probe-rs;latest;latest" \
-DTARGET_MCU=STM32H503RBTx -DINTEGRATION_TEST_BOARD=STM32H503xB -DUNITY_ROOT=<Unity>
cmake --build build/it-h503rb && ctest --test-dir build/it-h503rb

Probe selection (<Name>.Run, probe-rs list):

  1. Candidates: environment variable INTEGRATION_TEST_PROBE_SN (more equal boards), otherwise INTEGRATION_TEST_PROBE (probes of the board in the boards file by default), otherwise all connected probes. Probe is given by probe-rs selector (VID:PID[:SN]) or by serial number.
  2. With INTEGRATION_TEST_ID_ADDRESS the ID register (and flash size register) of every candidate is read - the first probe with matching MCU is used (the MCU of the board is always checked before download).
  3. Without MCU ID the first candidate is used - only one probe may be connected, if no candidate is given.
  4. The selected probe is locked until the firmware run ends (see Probe lock) - a candidate used by another client is not read. When no free candidate matches, the probe used by another client is waited for at most INTEGRATION_TEST_LOCK_TIMEOUT seconds.

Probe lock​

A debug probe is used by one client at a time - debugging, the integration tests of this host, the detection of connected boards and the jobs of the integration test pipeline on the IT agents (ItRun.cmake of EmBi_Tester) take the lock of the probe (IntegrationTesting_Lock.cmake). The lock is the lock of a file, the operating system releases it when its process ends - also when the process is killed or its SSH connection breaks, no lock stays behind.

ItemProtocol
Lock folderEnvironment variable INTEGRATION_TEST_LOCK_DIR, otherwise <temporary folder>/embi-probe-locks (/tmp on Linux and macOS, TEMP on Windows), mode 0777 (the clients run as different users).
Lock fileprobe-<key>.lock (mode 0666), <key> = serial number of the probe (VID:PID:SN or SN), VID-PID of a probe without a serial number; characters other than letters, digits, - and _ are replaced by _. The file is never removed.
Holder fileprobe-<key>.holder (mode 0666): <holder>, since <YYYY-MM-DD hh:mm:ss> UTC, written by the process that got the lock (<login>@<computer> (<purpose>), pipeline <definition> <run number> in a pipeline job).
Parent lockEnvironment variable INTEGRATION_TEST_PROBE_LOCKED - keys of the probes locked by a parent process (eg. ItRun.cmake for its whole ctest run), separated by ; or ,. Their lock is not requested again (the lock of a file belongs to the process).
ClientLock
<Name>.Run (IntegrationTesting_Run.cmake)The selected probe for the firmware run; a probe used by another client is waited for at most INTEGRATION_TEST_LOCK_TIMEOUT (default 600 s, environment variable of the same name overrides the host configuration), its holder is printed every minute.
Detection (IntegrationTesting_Detect.cmake)Every probe while its registers are read; a probe used by another client is not read, it is reported with its holder.
Debug server (IntegrationTesting_DebugServer.cmake, embi-agent debug-server of an IT agent, VS Code EmBi: Debug on board)The probe of the picked board for the whole debug session; a probe used by another client is not waited for - the session is not started and the holder is reported (a lock of the same holder, the previous session of a restart, is waited for 15 s).

Parameters:

ParameterDefaultDescription
INTEGRATION_TEST_BOARD-Board name, compile definition IT_BOARD_<name> selects pins in tests, parameters of the board from the boards file.
INTEGRATION_TEST_BOARDS_FILEBuild/IntegrationTestBoards.jsonBoards file - connected boards (detection).
INTEGRATION_TEST_MCU_PRESETS_FILEBsp/Ral/RalPresets.jsonRal presets with MCU identification.
INTEGRATION_TEST_PROBEprobes of the board / all probesProbes of the board - probe-rs selectors VID:PID[:SN] or serial numbers (list).
INTEGRATION_TEST_PROBE_SN-Environment variable at test time - probe (selector or serial number).
INTEGRATION_TEST_PROBE_RS_CHIPidentification device / TARGET_MCUprobe-rs chip - the first one probe-rs knows (probe-rs chip info), otherwise the board defines it (boards file). Names of MCUs are not interpreted.
INTEGRATION_TEST_ID_ADDRESSidentificationAddress of the MCU ID register, empty = MCU is not checked.
INTEGRATION_TEST_ID_MASKidentification / 0xFFFFFFFFMask of the ID bits.
INTEGRATION_TEST_ID_VALUEidentificationExpected ID (after mask).
INTEGRATION_TEST_FLASH_SIZE_ADDRESSidentificationAddress of the 16-bit flash size register [KB], empty = flash size is not checked.
INTEGRATION_TEST_FLASH_SIZE_KBidentificationExpected flash size [KB].
PROBE_RS_EXECUTABLEartifactprobe-rs (artifact probe-rs; probe-rs installed in the system is not used).
INTEGRATION_TEST_OUTPUT_SIZE4096Unity output buffer [B] (RAM of the smallest MCU - H503 has 32 KB).
INTEGRATION_TEST_TIMEOUT60Default timeout of all test cases of the firmware [s], TIMEOUT of the test overrides it.
INTEGRATION_TEST_CASE_TIMEOUT10Default timeout of one test case [s], CASE_TIMEOUT of the test overrides it.
INTEGRATION_TEST_CASE_HOST_TIME3Time of the host per test case [s], added to the timeout of all test cases (TIMEOUT + count of the test cases Γ— this time): one probe session per test case, about 1.5 s, a probe that does not open is opened again (about 4 s more).
INTEGRATION_TEST_LOCK_TIMEOUT600Longest wait for a probe used by another client [s] (Probe lock); environment variable of the same name at test time overrides it.
INTEGRATION_TEST_HOST-IT agent of the board: name of ItHosts.json or <user>@<host>[:<SSH port>], empty = this machine (Boards of IT agents); environment variable of the same name (SSH target) at test time overrides it.
INTEGRATION_TEST_OPTIMIZATION-O0Optimization option of the test firmware. Devices with a small flash need a smaller firmware, e.g. -Os for STM32H7R / STM32H7S (64 KB flash: Adc test set 76.7 KB at -O0, 41.3 KB at -Os). Set it in the hidden preset project of CMakePresets.json or by -DINTEGRATION_TEST_OPTIMIZATION=-Os.
INTEGRATION_TEST_RUN_KNOWN_DEFECTSOFFExecute tests marked by IT_KNOWN_DEFECT().

Useful:

  • ctest -L Gpio - tests of one module. A single function cannot be executed alone - the whole firmware runs once (<Name>.Run is added automatically as fixture).
  • IT_<Name>.elf can be debugged in any debugger as any firmware (-O0 -g3). Without host the firmware starts local session (all test cases, resets between them); hardware breakpoints are kept over system reset. The output is visible in integrationTesting_Mailbox.Output.

Requirements on the target​

The framework needs from the project (linker script, startup code, MCU configuration):

RequirementWhyFailure
Output section .noinit in the linker script (NOLOAD, in RAM, not zeroed or copied by the startup code)Result mailbox integrationTesting_Mailbox is placed by __attribute__((section(".noinit"))) - it keeps the session state and the output over system resets between test cases.Without the section the linker places the mailbox into another section or the startup code clears it: <Name>.Run fails with "mailbox is not kept over system reset" (session LOCL).
RAM of .noinit is not erased by system resetHardware erase of RAM on reset (eg. option bytes / reset configuration of the MCU) clears the mailbox the same way.Same as above.
RAM readable by the debug probe while the core runsHost polls the mailbox without stopping the test (probe-rs trace).Timeout of <Name>.Run.
Halt flag of the core readable by the firmware: DFSR.HALTED (Cortex-M4 / M7), DHCSR.S_RESTART_ST (Cortex-M33)Halt of the attach of the probe before a test case (session HTRC, ItCore/IntegrationTesting_Core.h).Another core has no handler: link error (IntegrationTesting_Core_WaitHalted).
GCC compatible toolchain__attribute__((section)), -Wl,-Map, objcopy -O ihex, size.Build error.

Example (GNU ld, .noinit after .bss):

.noinit (NOLOAD) :
{
. = ALIGN(4);
*(.noinit)
*(.noinit*)
. = ALIGN(4);
} >RAM

✍️ Writing tests​

Test sets are created from the template by Setup scripts (Setup.sh / Setup.bat: new module, or "Add tests to existing module" - HelperTools/Test_Handler). Module layout:

Bsp/Mcal/Gpio/
β”œβ”€β”€ CMakeLists.txt # + IntegrationTesting_AddPath(.../Tests/IntegrationTests)
└── Tests/
β”œβ”€β”€ UnitTests/ # host tests (UnitTesting)
└── IntegrationTests/ # this document
β”œβ”€β”€ CMakeLists.txt
β”œβ”€β”€ ItTest_Gpio.c # test cases
β”œβ”€β”€ ItTarget_Gpio.c # target interface of the test set
└── BspMain.h # entry point called by StartUp

Tests/IntegrationTests/CMakeLists.txt:

# StartUp calls BspMain() - the entry point of the test firmware is provided by the test set
if(NOT TARGET BspMain_Lib)
add_library(BspMain_Lib INTERFACE)
target_include_directories(BspMain_Lib INTERFACE ${CMAKE_CURRENT_SOURCE_DIR})
endif()

IntegrationTesting_Add_Test(
NAME Gpio
TEST_SOURCE ItTest_Gpio.c
SOURCES ItTarget_Gpio.c # target interface of the test set
LINK_LIBS StartUp_Lib # startup code (calls BspMain)
Gpio_Lib # module under test (+ its dependencies)
TIMEOUT 30 # all test cases [s]
CASE_TIMEOUT 5 # one test case [s]
)

Target interface (ItTarget_Gpio.c) - the framework calls only these two functions, everything else (including the entry point) is owned by the test set:

FunctionImplemented byDescription
entry point (eg. BspMain())test setCalled by the startup code, calls IntegrationTesting_Run() (never returns).
ItTarget_Init()test setEvery boot, before the test case: fault handlers, debug freeze of the peripherals used by the tests (probe-rs halts the core on every mailbox access), clocks, ...
ItTarget_SystemReset()test setReset of the target between the test cases (shall not return).
fault handlerstest setCollect registers and call IntegrationTesting_ReportFault( name, regs, cnt ) (never returns).

The template implements the interface for Cortex-M / STM32 (Nvic API, SCB fault status, SecureFault only on ARMv8-M main). Another architecture needs only another ItTarget_<Module>.c, the framework stays unchanged.

Test file:

#include "unity.h"
#include "IntegrationTesting.h"
#include "Gpio_Port.h"

#if defined(IT_BOARD_STM32H503xB) || \
defined(IT_BOARD_STM32H533xE)
#define IT_GPIO_FREE_PORT ( GPIO_PORT_A )
...
#else
#error "Board of Gpio integration tests is not defined (INTEGRATION_TEST_BOARD)."
#endif

void setUp( void ) { ... }
void tearDown( void ) { /* return backup domain / option bytes changes */ }

void It_Gpio_Set_PinPull_InputPullUp_PinReadsHigh( void ) { ... }

Test case expecting reset (eg. watchdog) - every expected reset executes the test case again with incremented stage. Reset source flags are target specific - the test case clears them itself:

void It_Iwdg_Timeout_ResetsMcu( void )
{
if( 0u == IntegrationTesting_Get_Stage() )
{
TEST_ASSERT_EQUAL( RCC_REQUEST_OK, Rcc_Set_ResetSourceClear() );
IntegrationTesting_Set_ResetExpected();
/* start watchdog, wait for reset */
TEST_FAIL_MESSAGE( "Reset did not occur" );
}
else
{
rcc_FlagState_t iwdgReset = RCC_FLAG_INACTIVE;
TEST_ASSERT_EQUAL( RCC_REQUEST_OK, Rcc_Get_ResetSource( RCC_RESET_SRC_IWDG, &iwdgReset ) );
TEST_ASSERT_EQUAL( RCC_FLAG_ACTIVE, iwdgReset );
}
}

Rules:

  • Test cases (ItTest_<Module>.c) test only through public API of the modules (<Mod>_Port.h) - no LL, no direct register access (MCAL layering). Dependencies are real modules, never mocks. Target specific code of the test set (debug freeze, fault registers) belongs to ItTarget_<Module>.c.
  • Every test case starts after system reset (core, peripherals, NVIC and clocks in reset state), so the order of tests does not matter. System reset does not reset the backup domain (RTC, LSE, TAMP backup registers) and option bytes - a test changing them returns them in tearDown().
  • Board dependent resources (pins, LEDs, loopback wiring) are selected by IT_BOARD_<name>, unknown board ends by #error. Tests needing external wiring document it in the file header.
  • Test names as in unit tests: test_<Function>_<Condition>_<ExpectedResult>.
  • Keep output small (INTEGRATION_TEST_OUTPUT_SIZE).

πŸ“Š Code coverage​

-DINTEGRATION_TEST_COVERAGE=ON measures which code of the tested modules the tests execute on the board:

  • the libraries under test (COVERAGE_LIBS of IntegrationTesting_Add_Test, default <NAME>_Lib) are compiled with the gcov instrumentation (--coverage -fprofile-info-section -fdata-sections),
  • the counters are kept over the resets between the test cases (linker scripts ItCore/IntegrationTesting_GcovCounters.ld and IntegrationTesting_GcovInfo.ld; they need the memory regions FLASH and RAM of the linker script),
  • after the last test case the firmware serializes the counters into the buffer integrationTesting_Coverage (size INTEGRATION_TEST_COVERAGE_SIZE, default 48 kB), <NAME>.Run reads it through the probe, stores it as IT_<NAME>.cov.ihex and converts it to .gcda files (ARM toolchain objcopy, gcov-tool merge-stream),
  • IntegrationTest_Coverage creates the report (<build>/Coverage; a plain ctest of such a build runs it after the last test and prints the table, INTEGRATION_TEST_COVERAGE_CTEST_REPORT, default ON), IntegrationTest_CoverageSummary prints the table, IntegrationTest_CoverageReset deletes the counters of earlier runs, IntegrationTest_CoverageMerge converts the stored data again.

The firmware gets bigger (libgcov, counters, buffer: tens of kB of FLASH and RAM) and slower, so it is a separate build option. A board host without the ARM toolchain (the board farm) only stores the data; the pipeline evaluates them in the stage Coverage. See Code coverage.


πŸ”­ Next steps​

  • Integration tests of other modules (Rcc, Exti, Nvic, Tim, Usart, Gpdma, Adc, I2c, Iwdg).
  • Artifact unity (see UnitTesting/README.md).
  • Renode: the same IT_<Name>.elf, mailbox read by Renode monitor (sysbus ReadDoubleWord).