kopia lustrzana https://github.com/espressif/esp-idf
Merge branch 'docs/test_linux' into 'master'
docs: refactored and updated Linux target docs See merge request espressif/esp-idf!21236pull/10469/head
Jakob Hasse
commit
58f58c25a6
|
|
@ -0,0 +1,128 @@
|
|||
Running Applications on Host
|
||||
============================
|
||||
|
||||
.. note::
|
||||
Running IDF applications on host is currently still an experimental feature, thus there is no guarantee for API stability. However, user feedback via the `ESP-IDF GitHub repository <https://github.com/espressif/esp-idf>` or the `ESP32 forum <https://esp32.com/>` is highly welcome, and may help influence the future of design of the IDF host-based applications.
|
||||
|
||||
This document provides an overview of the methods to run IDF applications on Linux, and what type of IDF applications can typically be run on Linux.
|
||||
|
||||
Introduction
|
||||
---------------------------
|
||||
|
||||
Typically, an IDF application is built (cross-compiled) on a host machine, uploaded (i.e., flashed) to an ESP chip for execution, and monitored by the host machine via a UART/USB port. However, execution of an IDF application on an ESP chip (hence forth referred to as "running on target") can be limiting in various development/usage/testing scenarios.
|
||||
|
||||
Therefore, it is possible for an IDF application to be built and executed entirely within the same Linux host machine (hence forth referred to as "running on host"). Running ESP-IDF applications on host has several advantages:
|
||||
|
||||
- No need to upload to a target.
|
||||
- Faster execution on a host machine, compared to running on an ESP chip.
|
||||
- No requirements for any specific hardware, except the host machine itself.
|
||||
- Easier automation and setup for software testing.
|
||||
- Large number of tools for code and runtime analysis (e.g. Valgrind).
|
||||
|
||||
A large number of IDF components depend on chip-specific hardware. These hardware dependencies must be mocked or simulated when running on host. ESP-IDF currently supports the following mocking and simulation approaches:
|
||||
|
||||
1. Using the `FreeRTOS POSIX/Linux simulator <https://www.freertos.org/FreeRTOS-simulator-for-Linux.html>`_ that simulates FreeRTOS scheduling. On top of this simulation, other APIs are also simulated or implemented when running on host.
|
||||
2. Using `CMock <https://www.throwtheswitch.org/cmock>`_ to mock all dependencies and run the code in complete isolation.
|
||||
|
||||
In principle, it is possible to mix both approaches (POSIX/Linux simulator and mocking using CMock), but this has not been done yet in ESP-IDF. Note that despite the name, the FreeRTOS POSIX/Linux simulator currently also works on MacOS. Running IDF applications on host machines is often used for testing. However, simulating the environment and mocking dependencies does not fully represent the target device. Thus, testing on the target device is still necessary, though with a different focus that usually puts more weight on integration and system testing.
|
||||
|
||||
.. note::
|
||||
Another possibility to run applications on the host is to use the QEMU simulator. However, QEMU development for IDF applications is currently work in progress and has not been documented yet.
|
||||
|
||||
CMock-Based Approach
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This approach uses the `CMock <https://www.throwtheswitch.org/cmock>`_ framework to solve the problem of missing hardware and software dependencies. CMock-based applications running on the host machine have the added advantage that they usually only compile the necessary code, i.e., the (mostly mocked) dependencies instead of the entire system. For a general introduction to Mocks and how to configure and use them in ESP-IDF, please refer to :ref:`mocks`.
|
||||
|
||||
|
||||
POSIX/Linux Simulator Approach
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The `FreeRTOS POSIX/Linux simulator <https://www.freertos.org/FreeRTOS-simulator-for-Linux.html>`_ is available on ESP-IDF as a preview target already. It is the base for the Linux target which is already available as a preview. Using this simulator, IDF components can be implemented on the host to make them available to IDF applications when running on host. Currently, only a limited number of components are ready to be built on Linux. Furthermore the functionality of each component ported to Linux may also be limited or different compared to the functionality when building that component for a chip target. For more information if the desired components are supported on Linux, please refer to :ref:`component-linux-mock-support`.
|
||||
|
||||
Requirements
|
||||
------------
|
||||
|
||||
.. include:: inc/linux-host-requirements.rst
|
||||
|
||||
If any mocks are used, then ``Ruby`` is required, too.
|
||||
|
||||
Build and Run
|
||||
-------------
|
||||
|
||||
To build the application on Linux, the target has to be set to ``linux`` and then it can be built and run:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
idf.py --preview set-target linux
|
||||
idf.py build
|
||||
idf.py monitor
|
||||
|
||||
.. _component-linux-mock-support:
|
||||
|
||||
Component Linux/Mock Support Overview
|
||||
-------------------------------------
|
||||
|
||||
Note that any "Yes" here does not necessarily mean a full implementation or mocking. It can also mean a partial implementation or mocking of functionality. Usually, the implementation or mocking is done to a point where enough functionality is provided to build and run a test application.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 20 10 10
|
||||
|
||||
* - Component
|
||||
- Mock
|
||||
- Simulation
|
||||
* - driver
|
||||
- Yes
|
||||
- No
|
||||
* - esp_common
|
||||
- No
|
||||
- Yes
|
||||
* - esp_event
|
||||
- Yes
|
||||
- Yes
|
||||
* - esp_hw_support
|
||||
- Yes
|
||||
- Yes
|
||||
* - esp_partition
|
||||
- Yes
|
||||
- No
|
||||
* - esp_rom
|
||||
- No
|
||||
- Yes
|
||||
* - esp_system
|
||||
- No
|
||||
- Yes
|
||||
* - esp_timer
|
||||
- Yes
|
||||
- No
|
||||
* - esp_tls
|
||||
- Yes
|
||||
- No
|
||||
* - freertos
|
||||
- Yes
|
||||
- Yes
|
||||
* - hal
|
||||
- No
|
||||
- Yes
|
||||
* - heap
|
||||
- No
|
||||
- Yes
|
||||
* - http_parser
|
||||
- Yes
|
||||
- No
|
||||
* - log
|
||||
- No
|
||||
- Yes
|
||||
* - lwip
|
||||
- Yes
|
||||
- No
|
||||
* - soc
|
||||
- No
|
||||
- Yes
|
||||
* - spi_flash
|
||||
- Yes
|
||||
- No
|
||||
* - tcp_transport
|
||||
- Yes
|
||||
- No
|
||||
|
|
@ -1,3 +1,13 @@
|
|||
- Installed IDF including all IDF requirements
|
||||
- CMock requirements (``Ruby``)
|
||||
- System package requirements (``libbsd``, ``libbsd-dev``)
|
||||
- A recent enough Linux or MacOS version and gcc compiler
|
||||
- All components the application depends on must be either supported on the Linux target (Linux/POSIX simulator) or mock-able
|
||||
|
||||
An application that runs on the Linux target has to set the ``COMPONENTS`` variable to ``main`` in the CMakeLists.txt of the application's root directory:
|
||||
|
||||
.. code-block:: cmake
|
||||
|
||||
set(COMPONENTS main)
|
||||
|
||||
|
||||
This prevents the automatic inclusion of all components from IDF to the build process which is otherwise done for convenience.
|
||||
|
|
|
|||
|
|
@ -41,7 +41,7 @@ API Guides
|
|||
thread-local-storage
|
||||
tools/index
|
||||
unit-tests
|
||||
linux-host-testing
|
||||
host-apps
|
||||
:SOC_USB_OTG_SUPPORTED: usb-otg-console
|
||||
:SOC_USB_SERIAL_JTAG_SUPPORTED: usb-serial-jtag-console
|
||||
:SOC_WIFI_SUPPORTED: wifi
|
||||
|
|
|
|||
|
|
@ -1,66 +0,0 @@
|
|||
Unit Testing on Linux
|
||||
=====================
|
||||
|
||||
.. note::
|
||||
Host testing with IDF is experimental for now. We try our best to keep interfaces stable but can't guarantee it for now. Feedback via github or the forum on esp32.com is highly welcome, though and may influence the future design of the host-based tests.
|
||||
|
||||
This article provides an overview of unit tests with IDF on Linux. For using unit tests on the target, please refer to :doc:`target based unit testing <unit-tests>`.
|
||||
|
||||
Embedded Software Tests
|
||||
-----------------------
|
||||
|
||||
Embedded software tests are challenging due to the following factors:
|
||||
|
||||
- Difficulties running tests efficiently.
|
||||
- Lack of many operating system abstractions when interfacing with hardware, making it difficult to isolate code under test.
|
||||
|
||||
To solve these two problems, Linux host-based tests with `CMock <https://www.throwtheswitch.org/cmock>`_ are introduced. Linux host-based tests are more efficient than unit tests on the target since they:
|
||||
|
||||
- Compile the necessary code only
|
||||
- Don't need time to upload to a target
|
||||
- Run much faster on a host-computer, compared to an ESP
|
||||
|
||||
Using the `CMock <https://www.throwtheswitch.org/cmock>`_ framework also solves the problem of hardware dependencies. Through mocking, hardware details are emulated and specified at run time, but only if necessary.
|
||||
|
||||
Of course, using code on the host and using mocks does not fully represent the target device. Thus, two kinds of tests are recommended:
|
||||
|
||||
1. Unit tests which test program logic on a Linux machine, isolated through mocks.
|
||||
2. System/Integration tests which test the interaction of components and the whole system. They run on the target, where irrelevant components and code may as well be emulated via mocks.
|
||||
|
||||
This documentation is about the first kind of tests. Refer to :doc:`target based unit testing <unit-tests>` for more information on target tests (the second kind of tests).
|
||||
|
||||
IDF Unit Tests on Linux Host
|
||||
----------------------------
|
||||
|
||||
The current focus of the Linux host tests is on creating isolated unit tests of components, while mocking the component's dependencies with CMock.
|
||||
|
||||
A complete implementation of IDF to run on Linux does not exist currently.
|
||||
|
||||
Examples for running IDF-built code on Linux host include (non-exhaustive list):
|
||||
|
||||
- :component_file:`unit test for the NVS Page class <nvs_flash/host_test/nvs_page_test/README.md>`.
|
||||
- :component_file:`unit test for esp_event <esp_event/host_test/esp_event_unit_test/main/esp_event_test.cpp>`.
|
||||
- :component_file:`unit test for mqtt <mqtt/host_test/README.md>`.
|
||||
|
||||
Inside the component which should be tested, there is a separate directory ``host_test``, besides the "traditional" ``test`` directory or the ``test_apps`` directory. It has one or more subdirectories::
|
||||
|
||||
- host_test/
|
||||
|
||||
- fixtures/
|
||||
contains test fixtures (structs/functions to do test case set-up and tear-down).
|
||||
If there are no fixtures, this can be ommitted.
|
||||
- <test_name>/
|
||||
IDF applications which run the tests
|
||||
- <test_name2>/
|
||||
Further tests are possible.
|
||||
|
||||
The IDF applications inside ``host_test`` set the mocking configuration as described in the :doc:`IDF unit test documentation <unit-tests>`.
|
||||
|
||||
The :component_file:`NVS page unit test <nvs_flash/host_test/nvs_page_test/main/nvs_page_test.cpp>` provides some illustration of how to control the mocks.
|
||||
|
||||
Requirements
|
||||
^^^^^^^^^^^^
|
||||
|
||||
.. include:: inc/linux-host-requirements.rst
|
||||
|
||||
The host tests have been tested on Ubuntu 20.04 with ``GCC`` version 9 and 10.
|
||||
|
|
@ -6,7 +6,7 @@ ESP-IDF provides the following methods to test software.
|
|||
|
||||
- Target based tests using a central unit test application which runs on the {IDF_TARGET_PATH_NAME}. These tests use the `Unity <https://www.throwtheswitch.org/unity>`_ unit test framework. They can be integrated into an ESP-IDF component by placing them in the component's ``test`` subdirectory. This document mainly introduces this target based tests.
|
||||
|
||||
- Linux-host based unit tests in which all the hardwares are abstracted via mocks. Linux-host based tests are still under development and only a small fraction of IDF components support them, currently. They are covered here: :doc:`Unit Testing on Linux <linux-host-testing>`.
|
||||
- Linux-host based unit tests in which part of the hardware can be abstracted via mocks. Currently, Linux-host based tests are still under development and only a small fraction of IDF components support them. More information on running IDF applications on the host can be found here: :doc:`Running Applications on the Host Machine <host-apps>`.
|
||||
|
||||
Normal Test Cases
|
||||
------------------
|
||||
|
|
@ -277,39 +277,43 @@ The cache compensated timer is an alternative to placing the code/data to be ben
|
|||
|
||||
One limitation of the cache compensated timer is that the task that benchmarked functions should be pinned to a core. This is due to each core having its own event counters that are independent of each other. For example, if ``ccomp_timer_start`` gets called on one core, put to sleep by the scheduler, wakes up, and gets rescheduled on the other core, then the corresponding ``ccomp_timer_stop`` will be invalid.
|
||||
|
||||
.. _mocks:
|
||||
|
||||
Mocks
|
||||
-----
|
||||
|
||||
.. note::
|
||||
Currently, mocking is only possible with some selected components when running on the Linux host. In the future, we plan to make essential components in IDF mockable. This will also include mocking when running on the {IDF_TARGET_NAME}.
|
||||
Currently, mocking is only possible with some selected components when running on the Linux host. In the future, we plan to make essential components in IDF mock-able. This will also include mocking when running on the {IDF_TARGET_NAME}.
|
||||
|
||||
One of the biggest problems regarding unit testing of embedded systems are the strong hardware dependencies. Running unit tests directly on the {IDF_TARGET_NAME} can be especially difficult for higher layer components for the following reasons:
|
||||
One of the biggest problems regarding unit testing on embedded systems are the strong hardware dependencies. Running unit tests directly on the {IDF_TARGET_NAME} can be especially difficult for higher layer components for the following reasons:
|
||||
|
||||
- Decreased test reliability due to lower layer components and/or hardware setup.
|
||||
- Increased difficulty in testing edge cases due to limitations of lower layer components and/or hardware setup
|
||||
- Increased difficulty in identifying the root cause due to the large number of dependencies influencing the behavior
|
||||
|
||||
When testing a particular component, (i.e., the component under test), software mocking allows the dependencies of the component under test to be substituted (i.e., mocked) entirely in software. To allow software mocking, ESP-IDF integrates the `CMock <https://www.throwtheswitch.org/cmock>`_ mocking framework as a component. With the addition of some CMake functions in the ESP-IDF's build system, it is possible to conveniently mock the entirety (or a part of) an IDF component.
|
||||
When testing a particular component, (i.e., the component under test), mocking allows the dependencies of the component under test to be substituted (i.e., mocked) entirely in software. Through mocking, hardware details are emulated and specified at run time, but only if necessary. To allow mocking, ESP-IDF integrates the `CMock <https://www.throwtheswitch.org/cmock>`_ mocking framework as a component. With the addition of some CMake functions in the ESP-IDF build system, it is possible to conveniently mock the entirety (or a part) of an IDF component.
|
||||
|
||||
Ideally, all components that the component under test is dependent on should be mocked, thus allowing the test environment complete control over all interactions with the component under test. However, if mocking all dependent components becomes too complex or too tedious (e.g. because you need to mock too many function calls) you have the following options:
|
||||
|
||||
.. list::
|
||||
- Include more "real" IDF code in the tests. This may work but increases the dependency on the "real" code's behavior. Furthermore, once a test fails, you may not know if the failure is in your actual code under tests or the "real" IDF code.
|
||||
- Re-evaluate the design of the code under test and attempt to reduce its dependencies by dividing the code under test into more manageable components. This may seem burdensome but it is common knowledge that unit tests often expose software design weaknesses. Fixing design weaknesses will not only help with unit testing in the short term, but will help future code maintenance as well.
|
||||
- Include more "real" IDF code in the tests. This may work but increases the dependency on the "real" code's behavior. Furthermore, once a test fails, you may not know if the failure is in your actual code under test or the "real" IDF code.
|
||||
- Re-evaluate the design of the code under test and attempt to reduce its dependencies by dividing the code under test into more manageable components. This may seem burdensome but it is quite common that unit tests expose software design weaknesses. Fixing design weaknesses will not only help with unit testing in the short term, but will help future code maintenance as well.
|
||||
|
||||
Refer to :component_file:`cmock/CMock/docs/CMock_Summary.md` for more details on how CMock works and how to create and use mocks.
|
||||
|
||||
Requirements
|
||||
^^^^^^^^^^^^
|
||||
The Linux target is the only target where mocking currently works. The following requirements are necessary to generate the mocks:
|
||||
|
||||
Mocking with CMock requires ``Ruby`` on the host machine. Furthermore, since mocking currently only works on the Linux target, the requirements of the latter also need to be fulfilled:
|
||||
|
||||
.. include:: inc/linux-host-requirements.rst
|
||||
|
||||
|
||||
Mock a Component
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
To create a mock version of a component, called a *component mock*, the component needs to be overwritten in a particular way. Overriding a component entails creating a component with the exact same name as the original component, then let the build system discover it later than the original component (see :ref:`Multiple components with the same name <cmake-components-same-name>` for more details).
|
||||
If a mocked component, called a *component mock*, is already available in ESP-IDF, then it can be used right away as long as it satisfies the required functionality. Refer to :ref:`component-linux-mock-support` to see which components are mocked already. Then refer to :ref:`adjustments_for_mocks` in order to use the component mock.
|
||||
|
||||
If IDF does not yet provide any component mock, it has to be created. To create a component mock, the component needs to be overwritten in a particular way. Overriding a component entails creating a component with the exact same name as the original component, then let the build system discover it later than the original component (see :ref:`Multiple components with the same name <cmake-components-same-name>` for more details).
|
||||
|
||||
In the component mock, the following parts are specified:
|
||||
|
||||
|
|
@ -341,7 +345,13 @@ For more details about the CMock configuration yaml file, have a look at :compon
|
|||
|
||||
Note that the component mock does not have to mock the original component in its entirety. As long as the test project's dependencies and dependencies of other code to the original components are satisfied by the component mock, partial mocking is adequate. In fact, most of the component mocks in IDF in ``tools/mocks`` are only partially mocking the original component.
|
||||
|
||||
Examples of component mocks can be found under :idf:`tools/mocks` in the IDF directory. General information on how to *override an IDF component* can be found in :ref:`Multiple components with the same name <cmake-components-same-name>`.
|
||||
Examples of component mocks can be found under :idf:`tools/mocks` in the IDF directory. General information on how to *override an IDF component* can be found in :ref:`Multiple components with the same name <cmake-components-same-name>`. There are several examples for testing code while mocking dependencies with CMock (non-exhaustive list):
|
||||
|
||||
- :component_file:`unit test for the NVS Page class <nvs_flash/host_test/nvs_page_test/README.md>`.
|
||||
- :component_file:`unit test for esp_event <esp_event/host_test/esp_event_unit_test/main/esp_event_test.cpp>`.
|
||||
- :component_file:`unit test for mqtt <mqtt/host_test/README.md>`.
|
||||
|
||||
.. _adjustments_for_mocks:
|
||||
|
||||
Adjustments in Unit Test
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
|
@ -354,4 +364,4 @@ The unit test needs to inform the cmake build system to mock dependent component
|
|||
|
||||
Both methods will override existing components in ESP-IDF with the component mock. The latter is particularly convenient if you use component mocks that are already supplied by IDF.
|
||||
|
||||
Users should refer to the ``esp_event`` host-based unit test and its :component_file:`esp_event/host_test/esp_event_unit_test/CMakeLists.txt` as an example of a component mock.
|
||||
Users can refer to the ``esp_event`` host-based unit test and its :component_file:`esp_event/host_test/esp_event_unit_test/CMakeLists.txt` as an example of a component mock.
|
||||
|
|
|
|||
|
|
@ -0,0 +1 @@
|
|||
.. include:: ../../en/api-guides/host-apps.rst
|
||||
|
|
@ -38,7 +38,7 @@ API 指南
|
|||
thread-local-storage
|
||||
tools/index
|
||||
unit-tests
|
||||
linux-host-testing
|
||||
host-apps
|
||||
:SOC_USB_OTG_SUPPORTED: usb-otg-console
|
||||
:SOC_USB_SERIAL_JTAG_SUPPORTED: usb-serial-jtag-console
|
||||
:SOC_WIFI_SUPPORTED: wifi
|
||||
|
|
|
|||
|
|
@ -1 +0,0 @@
|
|||
.. include:: ../../en/api-guides/linux-host-testing.rst
|
||||
|
|
@ -6,7 +6,7 @@ ESP-IDF 提供以下方法测试软件。
|
|||
|
||||
- 一种是基于目标的测试，该测试使用运行在 {IDF_TARGET_PATH_NAME} 上的中央单元测试应用程序。这些测试使用的是基于 `Unity <https://www.throwtheswitch.org/unity>`_ 的单元测试框架。通过把测试用例放在组件的 ``test`` 子目录，可以将其集成到 ESP-IDF 组件中。本文档主要介绍这种基于目标的测试方法。
|
||||
|
||||
- 另一种是基于 Linux 主机的单元测试，其中所有硬件行为都通过 Mock 组件进行模拟。此测试方法目前仍在开发中，暂且只有一小部分 IDF 组件支持 Mock，具体请参考 :doc:`基于 Linux 主机的单元测试 <linux-host-testing>`。
|
||||
- 另一种是基于 Linux 主机的单元测试，其中所有硬件行为都通过 Mock 组件进行模拟。此测试方法目前仍在开发中，暂且只有一小部分 IDF 组件支持 Mock，具体请参考 :doc:`基于 Linux 主机的单元测试 <host-apps>`。
|
||||
|
||||
添加常规测试用例
|
||||
----------------
|
||||
|
|
@ -277,6 +277,8 @@ DUT2（slave）终端::
|
|||
|
||||
缓存补偿定时器的限制之一是基准功能必须固定在一个内核上。这是由于每个内核都有自己的事件计数器，这些事件计数器彼此独立。例如，如果在一个内核上调用 ``ccomp_timer_start``，使调度器进入睡眠状态，唤醒并在在另一个内核上重新调度，那么对应的 ``ccomp_timer_stop`` 将无效。
|
||||
|
||||
.. _mocks:
|
||||
|
||||
Mocks
|
||||
----------
|
||||
|
||||
|
|
|
|||
Ładowanie…
Reference in New Issue