doc: linux host test documentation

pull/6974/head
Jakob Hasse 2020-12-24 17:34:03 +08:00
rodzic 469c137c83
commit 68393c41c4
11 zmienionych plików z 239 dodań i 36 usunięć

Wyświetl plik

@ -2,7 +2,7 @@ cmake_minimum_required(VERSION 3.5)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
set(COMPONENTS main)
idf_build_set_property(CONFIG_SPI_FLASH_MOCK 1)
idf_component_set_property(spi_flash USE_MOCK 1)
idf_build_set_property(COMPILE_DEFINITIONS "-DNO_DEBUG_STORAGE" APPEND)
project(host_nvs_page_test)

Wyświetl plik

@ -0,0 +1,105 @@
# NVS Page Unit Test
This unit test for the `nvs::Page` C++ class uses CMock to abstract its dependency to the `spi_flash` component. Its `CMakeLists.txt` builds mocks instead of the actual `spi_flash` component by setting the component property `USE_MOCK` for `spi_flash`. It also needs some stubs for CRC32 since the ROM component can not be mocked yet. These are retrieved from the `mock` directory inside `nvs_flash`.
The mocked version of `spi_flash` receives all function calls to the `esp_partition` API. E.g., have a look at the first unit test in the [NVS page test](main/nvs_page_test.cpp), the test function `test_Page_load_reading_header_fails()`. It simply checks whether an error is propagated correctly from the `esp_partition` API to upper layers during a call to `nvs::Page::load`. The relevant mocking code here is:
```c
esp_partition_read_raw_ExpectAnyArgsAndReturn(ESP_ERR_INVALID_ARG);
```
It tells the generated mocks to expect a call to `esp_partition` API's `esp_partition_read_raw()`, accept any arguments and return `ESP_ERR_INVALID_ARG`. When `nvs::Page::load` is called later, the mock will "replay" this action and the test will check that the return value is correctly propagated with:
```c
TEST_ASSERT_EQUAL(ESP_ERR_INVALID_ARG, page.load(&mock, 0));
```
For more information on how to control the mock's behavior, take a look at the [CMock README](../../../cmock/CMock/docs/CMock_Summary.md). The rest of `main/nvs_page_test.cpp` also provides many more examples on how to control the mocks.
## Requirements
* The usual IDF requirements
* The host's gcc/g++
* `ruby`
This application has been tested on Ubuntu 20.04 with `gcc` version *9.3.0*.
## Build
First, make sure that the target is set to Linux. Run `idf.py --preview set-target linux` if you are not sure. Then do a normal IDF build: `idf.py build`.
## Run
IDF monitor doesn't work yet for Linux. You have to run the app manually:
```bash
./build/host_nvs_page_test.elf
```
## Coverage
To generate the coverage, run: `idf.py coverage`. Afterwards, you can view the coverage by opening `build/coverage_report/index.html` with your browser. Note that you need to run the application at least once before generating the coverage information. If you run it multiple times, the coverage information adds up.
## Example Output
Ideally, all tests pass, which is indicated by the last two log lines after the dashed line:
```bash
build/host_nvs_page_test.elf
../main/nvs_page_test.cpp:880:test_Page_load_reading_header_fails:PASS
../main/nvs_page_test.cpp:881:test_Page_load_reading_data_fails:PASS
../main/nvs_page_test.cpp:882:test_Page_load__uninitialized_page_has_0xfe:PASS
../main/nvs_page_test.cpp:883:test_Page_load__initialized_corrupt_header:PASS
../main/nvs_page_test.cpp:884:test_Page_load_success:PASS
../main/nvs_page_test.cpp:885:test_Page_load_full_page:PASS
../main/nvs_page_test.cpp:886:test_Page_load__seq_number_0:PASS
../main/nvs_page_test.cpp:887:test_Page_erase__write_fail:PASS
../main/nvs_page_test.cpp:888:test_Page_erase__success:PASS
../main/nvs_page_test.cpp:889:test_Page_write__initialize_write_failure:PASS
../main/nvs_page_test.cpp:890:test_Page_write__write_data_fails:PASS
../main/nvs_page_test.cpp:891:test_page_write__write_correct_entry_state:PASS
../main/nvs_page_test.cpp:892:test_Page_write__write_correct_data:PASS
../main/nvs_page_test.cpp:893:test_Page_readItem__read_entry_fails:PASS
../main/nvs_page_test.cpp:894:test_Page_readItem__read_corrupted_entry:PASS
../main/nvs_page_test.cpp:895:test_Page_readItem__read_corrupted_second_read_fail:PASS
../main/nvs_page_test.cpp:896:test_Page_readItem__read_corrupted_erase_fail:PASS
../main/nvs_page_test.cpp:897:test_Page_readItem__read_entry_suceeds:PASS
../main/nvs_page_test.cpp:898:test_Page_readItem__blob_read_data_fails:PASS
../main/nvs_page_test.cpp:899:test_Page_readItem__blob_corrupt_data:PASS
../main/nvs_page_test.cpp:900:test_Page_readItem__blob_read_entry_suceeds:PASS
../main/nvs_page_test.cpp:901:test_Page_cmp__uninitialized:PASS
../main/nvs_page_test.cpp:902:test_Page_cmp__item_not_found:PASS
../main/nvs_page_test.cpp:903:test_Page_cmp__item_type_mismatch:PASS
../main/nvs_page_test.cpp:904:test_Page_cmp__item_content_mismatch:PASS
../main/nvs_page_test.cpp:905:test_Page_cmp__item_content_match:PASS
../main/nvs_page_test.cpp:906:test_Page_cmpItem__blob_data_mismatch:PASS
../main/nvs_page_test.cpp:907:test_Page_cmpItem__blob_data_match:PASS
../main/nvs_page_test.cpp:908:test_Page_eraseItem__uninitialized:PASS
../main/nvs_page_test.cpp:909:test_Page_eraseItem__key_not_found:PASS
../main/nvs_page_test.cpp:910:test_Page_eraseItem__write_fail:PASS
../main/nvs_page_test.cpp:911:test_Page_readItem__corrupt_data_erase_failure:PASS
../main/nvs_page_test.cpp:912:test_Page_eraseItem__write_succeed:PASS
../main/nvs_page_test.cpp:913:test_Page_findItem__uninitialized:PASS
../main/nvs_page_test.cpp:914:test_Page_find__wrong_ns:PASS
../main/nvs_page_test.cpp:915:test_Page_find__wrong_type:PASS
../main/nvs_page_test.cpp:916:test_Page_find__key_empty:PASS
../main/nvs_page_test.cpp:917:test_Page_find__wrong_key:PASS
../main/nvs_page_test.cpp:918:test_Page_find__too_large_index:PASS
../main/nvs_page_test.cpp:919:test_Page_findItem__without_read:PASS
../main/nvs_page_test.cpp:920:test_Page_markFull__wrong_state:PASS
../main/nvs_page_test.cpp:921:test_Page_markFreeing__wrong_state:PASS
../main/nvs_page_test.cpp:922:test_Page_markFull__success:PASS
../main/nvs_page_test.cpp:923:test_Page_markFreeing__success:PASS
../main/nvs_page_test.cpp:924:test_Page_markFull__write_fail:PASS
../main/nvs_page_test.cpp:925:test_Page_getVarDataTailroom__uninitialized_page:PASS
../main/nvs_page_test.cpp:926:test_Page_getVarDataTailroom__success:PASS
../main/nvs_page_test.cpp:927:test_Page_calcEntries__uninit:PASS
../main/nvs_page_test.cpp:928:test_Page_calcEntries__corrupt:PASS
../main/nvs_page_test.cpp:929:test_Page_calcEntries__active_wo_blob:PASS
../main/nvs_page_test.cpp:930:test_Page_calcEntries__active_with_blob:PASS
../main/nvs_page_test.cpp:931:test_Page_calcEntries__invalid:PASS
-----------------------
52 Tests 0 Failures 0 Ignored
OK
```

Wyświetl plik

@ -1,23 +0,0 @@
NVS Page Test for Host
======================
Build
-----
First, make sure that the target is set to linux.
Run ``idf.py --preview set-target linux`` to be sure.
Then do a normal IDF build: ``idf.py build``.
Run
---
IDF monitor doesn't work yet for Linux.
You have to run the app manually: ``./build/host_nvs_page_test.elf``.
Coverage
---
To generate the coverage, run: ``idf.py coverage``.
Afterwards, you can view the coverage by opening ``build/coverage_report/index.html`` with your browser.
Note that you need to run the application at least once before generating the coverage information.
If you run it multiple times, the coverage information adds up.

Wyświetl plik

@ -2,7 +2,6 @@ idf_component_register(SRCS "nvs_page_test.cpp"
INCLUDE_DIRS
"."
"${CMAKE_CURRENT_SOURCE_DIR}/../../fixtures"
"${CMAKE_CURRENT_SOURCE_DIR}/../../../test_nvs_host"
"${CMAKE_CURRENT_SOURCE_DIR}/../../../src"
REQUIRES cmock nvs_flash spi_flash)

Wyświetl plik

@ -1,4 +1,4 @@
idf_build_get_property(spi_flash_mock CONFIG_SPI_FLASH_MOCK)
idf_component_get_property(spi_flash_mock ${COMPONENT_NAME} USE_MOCK)
idf_build_get_property(target IDF_TARGET)
if(${spi_flash_mock})
message(STATUS "building SPI FLASH MOCKS")

Wyświetl plik

@ -38,7 +38,8 @@ API Guides
:SOC_ULP_SUPPORTED: ULP Coprocessor <ulp>
:esp32: ULP Coprocessor (Legacy GNU Make) <ulp-legacy>
:SOC_RISCV_COPROC_SUPPORTED: ULP-RISC-V Coprocessor <ulp-risc-v>
Unit Testing <unit-tests>
Unit Testing (Target) <unit-tests>
Unit Testing (Linux Host) <linux-host-testing>
:esp32: Unit Testing (Legacy GNU Make) <unit-tests-legacy>
:SOC_USB_SUPPORTED: USB Console <usb-console>
WiFi Driver <wifi>

Wyświetl plik

@ -0,0 +1,65 @@
Unit Testing on Linux
=====================
.. note::
Host testing with IDF is experimental for now. We try our best to keep interfaces stable but can't guarantee it for now. Feedback via github or the forum on esp32.com is highly welcome, though and may influence the future design of the host-based tests.
This article provides an overview of unit tests with IDF on Linux. For using unit tests on the target, please refer to :doc:`target based unit testing <unit-tests>`.
Embedded Software Tests
-----------------------
Embedded software tests are challenging due to the following factors:
- Difficulties running tests efficiently.
- Lack of many operating system abstractions when interfacing with hardware, making it difficult to isolate code under test.
To solve these two problems, Linux host-based tests with `CMock <https://www.throwtheswitch.org/cmock>`_ are introduced. Linux host-based tests are more efficient than unit tests on the target since they:
- Compile the necessary code only
- Don't need time to upload to a target
- Run much faster on a host-computer, compared to an ESP
Using the `CMock <https://www.throwtheswitch.org/cmock>`_ framework also solves the problem of hardware dependencies. Through mocking, hardware details are emulated and specified at run time, but only if necessary.
Of course, using code on the host and using mocks does not fully represent the target device. Thus, two kinds of tests are recommended:
1. Unit tests which test program logic on a Linux machine, isolated through mocks.
2. System/Integration tests which test the interaction of components and the whole system. They run on the target, where irrelevant components and code may as well be emulated via mocks.
This documentation is about the first kind of tests. Refer to :doc:`target based unit testing <unit-tests>` for more information on target tests (the second kind of tests).
IDF Unit Tests on Linux Host
----------------------------
The current focus of the Linux host tests is on creating isolated unit tests of components, while mocking the component's dependencies with CMock.
A complete implementation of IDF to run on Linux does not exist currently.
There are currently two examples for running IDF-built code on Linux host:
- An example :example_file:`hello-world application <build_system/cmake/linux_host_app/README.md>`
- A :component_file:`unit test for NVS <nvs_flash/host_test/nvs_page_test/README.md>`.
Inside the component which should be tested, there is a separate directory ``host_test``, besides the "traditional" ``test`` directory or the ``test_apps`` directory. It has one or more subdirectories::
- host_test/
- fixtures/
contains test fixtures (structs/functions to do test case set-up and tear-down).
If there are no fixtures, this can be ommitted.
- <test_name>/
IDF applications which run the tests
- <test_name2>/
Further tests are possible.
The IDF applications inside ``host_test`` set the mocking configuration as described in the :doc:`IDF unit test documentation <unit-tests>`.
The :component_file:`NVS page unit test <nvs_flash/host_test/nvs_page_test/main/nvs_page_test.cpp>` provides some illustration of how to control the mocks.
Requirements
^^^^^^^^^^^^
Besides the usual IDF requirements and CMock requirements, only the host's ``GCC/g++`` is required.
The host tests have been tested on Ubuntu 20.04 with ``GCC`` version 9 and 10.

Wyświetl plik

@ -2,7 +2,10 @@ Unit Testing in {IDF_TARGET_NAME}
=================================
:link_to_translation:`zh_CN:[中文]`
ESP-IDF comes with a unit test application that is based on the Unity - unit test framework. Unit tests are integrated in the ESP-IDF repository and are placed in the ``test`` subdirectories of each component respectively.
ESP-IDF comes with two possibilities to test software.
- A unit test application which runs on the target and that is based on the Unity - unit test framework. These unit tests are integrated in the ESP-IDF repository and are placed in the ``test`` subdirectories of each component respectively. Target-based unit tests are covered in this document.
- Linux-host based unit tests in which all the hardware is abstracted via mocks. Linux-host based tests are still under development and only a small fraction of IDF components supports them currently. They are covered here: :doc:`target based unit testing <linux-host-testing>`.
Normal Test Cases
------------------
@ -274,13 +277,24 @@ One limitation of the cache compensated timer is that the task that benchmarked
Mocks
-----
ESP-IDF has a component which integrates the CMock mocking framework. CMock usually uses Unity as a submodule, but due to some Espressif-internal limitations with CI, we still have Unity as an ordinary module in ESP-IDF.
One of the biggest problems for unit testing in embedded systems are the strong hardware dependencies. This is why ESP-IDF has a component which integrates the `CMock <https://www.throwtheswitch.org/cmock>`_ mocking framework. Ideally, all components other than the one which should be tested *(component under test)* are mocked. This way, the test environment has complete control over all the interaction with the component under test. However, if mocking becomes problematic due to the tests becoming too specific, more "real" IDF code can always be included into the tests.
To use the IDF-supplied Unity component which isn't a submodule, the build system needs to pass an environment variable ``UNITY_IDR`` to CMock. This variable simply contains the path to the Unity directory in IDF, e.g. ``export "UNITY_DIR=${IDF_PATH}/components/unity/unity"``.
Besides the usual IDF requirements, ``ruby`` is necessary to generate the mocks. Refer to :component_file:`cmock/CMock/docs/CMock_Summary.md` for more details on how CMock works and how to create and use mocks.
Refer to :component_file:`cmock/CMock/lib/cmock_generator.rb` to see how the Unity directory is determined in CMock.
In IDF, adjustments are necessary inside the component(s) that should be mocked as well as inside the unit test, compared to writing normal components or unit tests without mocking.
An example cmake build command to create mocks of a component inside that component's CMakeLists.txt may look like this:
Adjustments in Mock Component
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The component that should be mocked requires a separate ``mock`` directory containing all additional files needed specifically for the mocking. Most importantly, it contains ``mock_config.yaml`` which configures CMock. For more details on what the options inside that configuration file mean and how to write your own, please take a look at the :component_file:`CMock documentation <cmock/CMock/docs/CMock_Summary.md>`. It may be necessary to have some more files related to mocking which should also be placed inside the `mock` directory.
Furthermore, the component's ``CMakeLists.txt`` needs a switch to build mocks instead of the actual code. This is usually done by checking the component property ``USE_MOCK`` for the particular component. E.g., the ``spi_flash`` component execute the following code in its ``CMakeLists.txt`` to check whether mocks should be built:
.. code-block:: cmake
idf_component_get_property(spi_flash_mock ${COMPONENT_NAME} USE_MOCK)
An example CMake build command to create mocks of a component inside its ``CMakeLists.txt`` may look like this:
.. code-block:: cmake
@ -290,6 +304,19 @@ An example cmake build command to create mocks of a component inside that compon
COMMAND ${CMAKE_COMMAND} -E env "UNITY_DIR=${IDF_PATH}/components/unity/unity" ruby ${CMOCK_DIR}/lib/cmock.rb -o${CMAKE_CURRENT_SOURCE_DIR}/mock/mock_config.yaml ${MOCK_HEADERS}
)
${MOCK_OUTPUT} contains all CMock generated output files, ${MOCK_HEADERS} contains all headers to be mocked and ${CMOCK_DIR} needs to be set to CMock directory inside IDF. ${CMAKE_COMMAND} is automatically set.
``${MOCK_OUTPUT}`` contains all CMock generated output files, ``${MOCK_HEADERS}`` contains all headers to be mocked and ``${CMOCK_DIR}`` needs to be set to the CMock directory inside IDF. ``${CMAKE_COMMAND}`` is automatically set by the IDF build system.
Refer to :component_file:`cmock/CMock/docs/CMock_Summary.md` for more details on how CMock works and how to create and use mocks.
One aspect of CMock's usage is special here: CMock usually uses Unity as a submodule, but due to some Espressif-internal limitations with CI, IDF still uses Unity as an ordinary module in ESP-IDF. To use the IDF-supplied Unity component, which isn't a submodule, the build system needs to pass an environment variable ``UNITY_IDR`` to CMock. This variable simply contains the path to the Unity directory in IDF, e.g. ``export "UNITY_DIR=${IDF_PATH}/components/unity/unity"``. Refer to :component_file:`cmock/CMock/lib/cmock_generator.rb` to see how the Unity directory is determined in CMock.
An example ``CMakeLists.txt`` which enables mocking exists :component_file:`in spi_flash <spi_flash/CMakeLists.txt>`
Adjustments in Unit Test
^^^^^^^^^^^^^^^^^^^^^^^^
The unit test needs to set the component property ``USE_MOCK`` for the component that should be mocked. This lets the dependent component build the mocks instead of the actual component. E.g., in the nvs host test's :component_file:`CMakeLists.txt <nvs_flash/host_test/nvs_page_test/CMakeLists.txt>`, ``spi_flash`` mocks are enabled by the following line:
.. code-block:: cmake
idf_component_set_property(spi_flash USE_MOCK 1)
Refer to the :component_file:`NVS host unit test <nvs_flash/host_test/nvs_page_test/README.md>` for more information on how to use and control CMock inside a unit test.

Wyświetl plik

@ -38,7 +38,8 @@ API 指南
:SOC_ULP_SUPPORTED: ULP 协处理器 <ulp>
:esp32: ULP (传统 GNU Make) <ulp-legacy>
:esp32s2: ULP-RISC-V 协处理器 <ulp-risc-v>
单元测试 <unit-tests>
单元测试 (Target) <unit-tests>
单元测试 (Linux Host) <linux-host-testing>
:esp32: 单元测试 (传统 GNU Make) <unit-tests-legacy>
:esp32s2: USB 控制台 <usb-console>
Wi-Fi 驱动 <wifi>

Wyświetl plik

@ -0,0 +1 @@
.. include:: ../../en/api-guides/linux-host-testing.rst

Wyświetl plik

@ -1,8 +1,35 @@
| Supported Targets | Linux |
| ----------------- | ----- |
This hello-world example builds a simple hello-world application for Linux.
The compiler used is the Linux-gcc.
There are two major differences to an IDF application built for an ESP chip compared to an application build for Linux:
1. The entry-point on Linux is `int main(int argc, char **argv)`, instead of `void app_main(void)` on an ESP chip.
In this example for Linux, the `void app_main(void)` function is still included to make the connection to the IDF entry point clearer.
However, it is simply called by `int main(int argc, char **argv)`.
Refer to the source file [linux_host_app.cpp](main/linux_host_app.cpp) to see how it is used.
2. The project-level [CMakeLists.txt](CMakeLists.txt) for Linux is different from that of a normal IDF application for an ESP chip.
On Linux, there is an additional line `set(COMPONENTS main)`, which clears the common requirements (default dependencies usually included in all IDF applications).
This is currently necessary as the Linux-host feature is still under developement.
Otherwise, a lot of hardware-dependent code would be pulled in.
# Build
`idf.py build` (sdkconfig.defaults sets the linux target by default)
Source the IDF environment as usual, then set the Linux target:
```bash
idf.py --preview set-target linux
```
sdkconfig.defaults sets the Linux target by default, so this not strictly necessary.
Once this is done, build the application:
```bash
idf.py build
```
Since this application runs on host, the flashing step is unnecessary.
# Run
```bash
`build/linux_host_app.elf`
```