kopia lustrzana https://github.com/espressif/esp-idf
docs: fix the current doc section level and outdated content
Fu Hanxi
rodzic
b7a2d77e59
commit
c3c48b95da
|
|
@ -1,3 +1,4 @@
|
|||
===============================
|
||||
ESP-IDF Tests with Pytest Guide
|
||||
===============================
|
||||
|
||||
|
|
@ -10,7 +11,7 @@ This documentation is a guide that introduces the following aspects:
|
|||
5. Tips and tricks on pytest
|
||||
|
||||
Disclaimer
|
||||
----------
|
||||
==========
|
||||
|
||||
In ESP-IDF, we use the following plugins by default:
|
||||
|
||||
|
|
@ -20,15 +21,15 @@ In ESP-IDF, we use the following plugins by default:
|
|||
All the introduced concepts and usages are based on the default behavior in ESP-IDF. Not all of them are available in vanilla pytest.
|
||||
|
||||
Installation
|
||||
------------
|
||||
============
|
||||
|
||||
All dependencies could be installed by running the install script with the ``--enable-pytest`` argument, e.g. ``$ install.sh --enable-pytest``.
|
||||
All dependencies could be installed by running the install script with the ``--enable-pytest`` argument, e.g., ``$ install.sh --enable-pytest``.
|
||||
|
||||
Common Issues During Installation
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
---------------------------------
|
||||
|
||||
No Package 'dbus-1' found
|
||||
"""""""""""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If you're facing an error message like:
|
||||
|
||||
|
|
@ -41,7 +42,7 @@ If you're facing an error message like:
|
|||
Consider adjusting the PKG_CONFIG_PATH environment variable if you
|
||||
installed software in a non-standard prefix.
|
||||
|
||||
If you're running a ubuntu system, you may need to run:
|
||||
If you're running under ubuntu system, you may need to run:
|
||||
|
||||
.. code:: shell
|
||||
|
||||
|
|
@ -56,7 +57,7 @@ or
|
|||
For other linux distros, you may Google the error message and find the solution. This issue could be solved by installing the related header files.
|
||||
|
||||
Invalid command 'bdist_wheel'
|
||||
"""""""""""""""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If you're facing an error message like:
|
||||
|
||||
|
|
@ -79,10 +80,10 @@ Or
|
|||
Before running the pip commands, please make sure you're using the IDF python virtual environment.
|
||||
|
||||
Basic Concepts
|
||||
--------------
|
||||
==============
|
||||
|
||||
Component-based Unit Tests
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
--------------------------
|
||||
|
||||
Component-based unit tests are our recommended way to test your component. All the test apps should be located under ``${IDF_PATH}/components/<COMPONENT_NAME>/test_apps``.
|
||||
|
||||
|
|
@ -112,7 +113,7 @@ For example:
|
|||
└── CMakeLists.txt
|
||||
|
||||
Example Tests
|
||||
^^^^^^^^^^^^^
|
||||
-------------
|
||||
|
||||
Example Tests are tests for examples that are intended to demonstrate parts of the ESP-IDF functionality to our customers.
|
||||
|
||||
|
|
@ -131,17 +132,17 @@ For example:
|
|||
└── pytest_example_1.py
|
||||
|
||||
Custom Tests
|
||||
^^^^^^^^^^^^
|
||||
------------
|
||||
|
||||
Custom Tests are tests that aim to run some arbitrary test internally. They are not intended to demonstrate the ESP-IDF functionality to our customers in any way.
|
||||
|
||||
All the test apps should be located under ``${IDF_PATH}/tools/test_apps``. For more information please refer to the :idf_file:`Custom Test Readme <tools/test_apps/README.md>`.
|
||||
|
||||
Pytest in ESP-IDF
|
||||
-----------------
|
||||
=================
|
||||
|
||||
Pytest Execution Process
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
------------------------
|
||||
|
||||
1. Bootstrapping Phase
|
||||
|
||||
|
|
@ -199,8 +200,8 @@ Pytest Execution Process
|
|||
|
||||
Combine the junit reports if the junit reports of the unity test cases are generated.
|
||||
|
||||
Example Code
|
||||
^^^^^^^^^^^^
|
||||
Getting Started Example
|
||||
-----------------------
|
||||
|
||||
This code example is taken from :idf_file:`pytest_console_basic.py <examples/system/console/basic/pytest_console_basic.py>`.
|
||||
|
||||
|
|
@ -219,31 +220,22 @@ This code example is taken from :idf_file:`pytest_console_basic.py <examples/sys
|
|||
elif config == 'nohistory':
|
||||
dut.expect('Command history disabled')
|
||||
|
||||
.. note::
|
||||
|
||||
Using ``expect_exact`` is better here. For further reading about the different types of ``expect`` functions, please refer to the `pytest-embedded Expecting documentation <https://docs.espressif.com/projects/pytest-embedded/en/latest/expecting>`__.
|
||||
Let's go through this simple test case line by line in the following subsections.
|
||||
|
||||
Use Markers to Specify the Supported Targets
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
You can use markers to specify the supported targets and the test env in CI. You can run ``pytest --markers`` to get more details about different markers.
|
||||
|
||||
.. code:: python
|
||||
|
||||
@pytest.mark.esp32 # <-- support esp32
|
||||
@pytest.mark.esp32c3 # <-- support esp32c3
|
||||
@pytest.mark.generic # <-- test env `generic, would assign to runner with tag `generic`
|
||||
@pytest.mark.generic # <-- test env "generic"
|
||||
|
||||
Besides, if the test case supports all officially ESP-IDF-supported targets, like esp32, esp32s2, esp32s3, esp32c3 for now (2022.2), you can use a special marker ``supported_targets`` to apply them all in one line.
|
||||
The above lines indicate that this test case supports target esp32 and esp32c3, the target board type should be "generic". If you want to know what is the "generic" type refers to, you may run ``pytest --markers`` to get the detailed information of all markers.
|
||||
|
||||
This code example is taken from :idf_file:`pytest_gptimer_example.py <examples/peripherals/timer_group/gptimer/pytest_gptimer_example.py>`.
|
||||
.. note::
|
||||
|
||||
.. code:: python
|
||||
|
||||
@pytest.mark.supported_targets
|
||||
@pytest.mark.generic
|
||||
def test_gptimer_example(dut: IdfDut) -> None:
|
||||
...
|
||||
If the test case supports all officially ESP-IDF supported targets (You may check the value via "idf.py --list-targets"), you can use a special marker ``supported_targets`` to apply all of them in one line.
|
||||
|
||||
Use Params to Specify the sdkconfig Files
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
|
@ -264,11 +256,26 @@ Overall, this test function would be replicated to 4 test cases:
|
|||
- esp32c3.history.test_console_advanced
|
||||
- esp32c3.nohistory.test_console_advanced
|
||||
|
||||
Expect From the Serial output
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: python
|
||||
|
||||
def test_console_advanced(config: str, dut: IdfDut) -> None: # The value of argument ``config`` is assigned by the parametrization.
|
||||
if config == 'history':
|
||||
dut.expect('Command history enabled')
|
||||
elif config == 'nohistory':
|
||||
dut.expect('Command history disabled')
|
||||
|
||||
When we're using ``dut.expect(...)``, the string would be compiled into regex at first, and then seeks through the serial output until the compiled regex is matched, or a timeout is exceeded. You may have to pay extra attention when the string contains regex keyword characters, like parentheses, or square brackets.
|
||||
|
||||
Actually using ``dut.expect_exact(...)`` here is better, since it would seek until the string is matched. For further reading about the different types of ``expect`` functions, please refer to the `pytest-embedded Expecting documentation <https://docs.espressif.com/projects/pytest-embedded/en/latest/expecting>`__.
|
||||
|
||||
Advanced Examples
|
||||
^^^^^^^^^^^^^^^^^
|
||||
-----------------
|
||||
|
||||
Multi Dut Tests with the Same App
|
||||
"""""""""""""""""""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: python
|
||||
|
||||
|
|
@ -286,7 +293,7 @@ Multi Dut Tests with the Same App
|
|||
After setting the param ``count`` to 2, all these fixtures are changed into tuples.
|
||||
|
||||
Multi Dut Tests with Different Apps
|
||||
"""""""""""""""""""""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This code example is taken from :idf_file:`pytest_wifi_getting_started.py <examples/wifi/getting_started/pytest_wifi_getting_started.py>`.
|
||||
|
||||
|
|
@ -312,7 +319,7 @@ Here the first dut was flashed with the app :idf_file:`softap <examples/wifi/get
|
|||
Here the ``app_path`` should be set with absolute path. the ``__file__`` macro in python would return the absolute path of the test script itself.
|
||||
|
||||
Multi Dut Tests with Different Apps, and Targets
|
||||
""""""""""""""""""""""""""""""""""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This code example is taken from :idf_file:`pytest_wifi_getting_started.py <examples/wifi/getting_started/pytest_wifi_getting_started.py>`. As the comment says, for now it's not running in the ESP-IDF CI.
|
||||
|
||||
|
|
@ -340,7 +347,7 @@ Overall, this test function would be replicated to 2 test cases:
|
|||
- softap with esp32s2 target, and station with esp32 target
|
||||
|
||||
Support different targets with different sdkconfig files
|
||||
""""""""""""""""""""""""""""""""""""""""""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This code example is taken from :idf_file:`pytest_panic.py <tools/test_apps/system/panic/pytest_panic.py>` as an advanced example.
|
||||
|
||||
|
|
@ -359,7 +366,7 @@ This code example is taken from :idf_file:`pytest_panic.py <tools/test_apps/syst
|
|||
...
|
||||
|
||||
Use Custom Class
|
||||
""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
Usually, you can write a custom class in these conditions:
|
||||
|
||||
|
|
@ -389,7 +396,7 @@ This code example is taken from :idf_file:`panic/conftest.py <tools/test_apps/sy
|
|||
``replace_dut_class`` is a `module-scoped <https://docs.pytest.org/en/latest/how-to/fixtures.html#scope-sharing-fixtures-across-classes-modules-packages-or-session>`__ `autouse <https://docs.pytest.org/en/latest/how-to/fixtures.html#autouse-fixtures-fixtures-you-don-t-have-to-request>`__ fixture. This function replaces the ``IdfDut`` class with your custom class.
|
||||
|
||||
Mark Flaky Tests
|
||||
""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
Sometimes, our test is based on ethernet or wifi. The network may cause the test flaky. We could mark the single test case within the code repo.
|
||||
|
||||
|
|
@ -404,7 +411,7 @@ This code example is taken from :idf_file:`pytest_esp_eth.py <components/esp_eth
|
|||
This flaky marker means that if the test function failed, the test case would rerun for a maximum of 3 times with 5 seconds delay.
|
||||
|
||||
Mark Known Failure Cases
|
||||
""""""""""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Sometimes a test couldn't pass for the following reasons:
|
||||
|
||||
|
|
@ -423,7 +430,7 @@ This code example is taken from :idf_file:`pytest_panic.py <tools/test_apps/syst
|
|||
This marker means that if the test would be a known failure one on esp32s2.
|
||||
|
||||
Mark Nightly Run Test Cases
|
||||
"""""""""""""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Some test cases are only triggered in nightly run pipelines due to a lack of runners.
|
||||
|
||||
|
|
@ -434,7 +441,7 @@ Some test cases are only triggered in nightly run pipelines due to a lack of run
|
|||
This marker means that the test case would only be run with env var ``NIGHTLY_RUN`` or ``INCLUDE_NIGHTLY_RUN``.
|
||||
|
||||
Mark Temp Disabled in CI
|
||||
""""""""""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Some test cases which can pass locally may need to be temporarily disabled in CI due to a lack of runners.
|
||||
|
||||
|
|
@ -445,7 +452,7 @@ Some test cases which can pass locally may need to be temporarily disabled in CI
|
|||
This marker means that the test case could still be run locally with ``pytest --target esp32``, but will not run in CI.
|
||||
|
||||
Run Unity Test Cases
|
||||
""""""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
For component-based unit test apps, one line could do the trick to run all single-board test cases, including normal test cases and multi-stage test cases:
|
||||
|
||||
|
|
@ -477,22 +484,22 @@ You may also see that there are some test scripts with the following statements,
|
|||
For further reading about our unit testing in ESP-IDF, please refer to :doc:`our unit testing guide <../api-guides/unit-tests>`.
|
||||
|
||||
Run the Tests in CI
|
||||
-------------------
|
||||
===================
|
||||
|
||||
The workflow in CI is simple, build jobs -> target test jobs.
|
||||
|
||||
Build Jobs
|
||||
^^^^^^^^^^
|
||||
----------
|
||||
|
||||
Build Job Names
|
||||
"""""""""""""""
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
- Component-based Unit Tests: ``build_pytest_components_<target>``
|
||||
- Example Tests: ``build_pytest_examples_<target>``
|
||||
- Custom Tests: ``build_pytest_test_apps_<target>``
|
||||
|
||||
Build Job Command
|
||||
"""""""""""""""""
|
||||
Build Job Commands
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The command used by CI to build all the relevant tests is: ``python $IDF_PATH/tools/ci/ci_build_apps.py <parent_dir> --target <target> -vv --pytest-apps``
|
||||
|
||||
|
|
@ -515,17 +522,17 @@ For example, If you run ``python $IDF_PATH/tools/ci/ci_build_apps.py $IDF_PATH/e
|
|||
All the binaries folders would be uploaded as artifacts under the same directories.
|
||||
|
||||
Target Test Jobs
|
||||
^^^^^^^^^^^^^^^^
|
||||
----------------
|
||||
|
||||
Target Test Job Names
|
||||
"""""""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- Component-based Unit Tests: ``component_ut_pytest_<target>_<test_env>``
|
||||
- Example Tests: ``example_test_pytest_<target>_<test_env>``
|
||||
- Custom Tests: ``test_app_test_pytest_<target>_<test_env>``
|
||||
|
||||
Target Test Job Command
|
||||
"""""""""""""""""""""""
|
||||
Target Test Job Commands
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The command used by CI to run all the relevant tests is: ``pytest <parent_dir> --target <target> -m <test_env_marker>``
|
||||
|
||||
|
|
@ -534,7 +541,7 @@ All test cases with the specified target marker and the test env marker under th
|
|||
The binaries in the target test jobs are downloaded from build jobs, the artifacts would be placed under the same directories.
|
||||
|
||||
Run the Tests Locally
|
||||
---------------------
|
||||
=====================
|
||||
|
||||
The local executing process is the same as the CI process.
|
||||
|
||||
|
|
@ -550,10 +557,10 @@ For example, if you want to run all the esp32 tests under the ``$IDF_PATH/exampl
|
|||
$ pytest --target esp32
|
||||
|
||||
Tips and Tricks
|
||||
---------------
|
||||
===============
|
||||
|
||||
Filter the Test Cases
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
---------------------
|
||||
|
||||
- filter by target with ``pytest --target <target>``
|
||||
|
||||
|
|
@ -566,52 +573,40 @@ Filter the Test Cases
|
|||
In other cases, pytest would run all the test cases with sdkconfig file ``sdkconfig.ci.<sdkconfig>``.
|
||||
|
||||
Add New Markers
|
||||
^^^^^^^^^^^^^^^
|
||||
---------------
|
||||
|
||||
We’re using two types of custom markers, target markers which indicate that the test cases should support this target, and env markers which indicate that the test case should be assigned to runners with these tags in CI.
|
||||
|
||||
You can add new markers by adding one line under the ``${IDF_PATH}/conftest.py``. If it's a target marker, it should be added into ``TARGET_MARKERS``. If it's a marker that specifies a type of test environment, it should be added into ``ENV_MARKERS``. The grammar should be: ``<marker_name>: <marker_description>``.
|
||||
|
||||
Generate JUnit Report
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
---------------------
|
||||
|
||||
You can call pytest with ``--junitxml <filepath>`` to generate the JUnit report. In ESP-IDF, the test case name would be unified as "<target>.<config>.<function_name>".
|
||||
|
||||
Skip Auto Flash Binary
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
----------------------
|
||||
|
||||
Skipping auto-flash binary every time would be useful when you're debugging your test script.
|
||||
|
||||
You can call pytest with ``--skip-autoflash y`` to achieve it.
|
||||
|
||||
Record Statistics
|
||||
^^^^^^^^^^^^^^^^^
|
||||
-----------------
|
||||
|
||||
Sometimes you may need to record some statistics while running the tests, like the performance test statistics.
|
||||
|
||||
You can use `record_xml_attribute <https://docs.pytest.org/en/latest/how-to/output.html?highlight=junit#record-xml-attribute>`__ fixture in your test script, and the statistics would be recorded as attributes in the JUnit report.
|
||||
|
||||
Logging System
|
||||
^^^^^^^^^^^^^^
|
||||
--------------
|
||||
|
||||
Sometimes you may need to add some extra logging lines while running the test cases.
|
||||
|
||||
You can use `python logging module <https://docs.python.org/3/library/logging.html>`__ to achieve this.
|
||||
|
||||
Known Limitations and Workarounds
|
||||
---------------------------------
|
||||
|
||||
Avoid Using ``Thread`` for Performance Test
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
``pytest-embedded`` is using some threads internally to help gather all stdout to the pexpect process. Due to the limitation of `Global Interpreter Lock <https://en.wikipedia.org/wiki/Global_interpreter_lock>`__, if you're using threads to do performance tests, these threads would block each other and there would be great performance loss.
|
||||
|
||||
**workaround**
|
||||
|
||||
Use `Process <https://docs.python.org/3/library/multiprocessing.html#the-process-class>`__ instead, the APIs should be almost the same as ``Thread``.
|
||||
|
||||
Further Readings
|
||||
----------------
|
||||
================
|
||||
|
||||
- pytest documentation: https://docs.pytest.org/en/latest/contents.html
|
||||
- pytest-embedded documentation: https://docs.espressif.com/projects/pytest-embedded/en/latest/
|
||||
|
|
|
|||
Ładowanie…
Reference in New Issue