1. The original test, before hint modules support was added, used
tempfile.NamedTemporaryFile in a way which is not supported on windows.
It was having the file open, which the hints tried to read it, leading
the EPERM exception. The docs[1] says this is not supported.
<quote>
Whether the name can be used to open the file a second time, while the
named temporary file is still open, varies across platforms (it can be
so used on Unix; it cannot on Windows)
</quote>
2. The hint module component_requirements test used the idf.py directly,
which is idf.py.exe on windows.Now it's starting idf.py through python.
We could probably used shell=True, but this approach is used in other
tests too.
Anyway the test are now passing on windows.
[1] https://docs.python.org/3/library/tempfile.html
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Following tests were added.
1. Test for missing header directory in component's INCLUDE_DIRS
2. Test for missing dependency in component's PRIV_REQUIRES
3. Test for missing dependency in component's REQUIRES
4. Test for dependency in PRIV_REQUIRES which should be in REQUIRES
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Implementation of hint module for component dependency. It can provide
hint about missing header directory in component's INCLUDE_DIRS or
about missing component dependency in REQUIRES or PRIV_REQUIRES.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently hints are supported based on hints.yml only, which may be
limiting for some use cases. This introduces a generic plugin approach,
which allows to implement hint module that doesn't require entry in hints.yml.
Such module has the full command output available and it is not limited to
a single regex in hints.yml.
Note that regex in hint.yml expects the output concatenated into a single line,
but hint modules are getting the output unchanged.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Introduces get_build_context() helper, which allows to get build context, e.g.
project description, at places where this info is not available. The
build context is set within ensure_build_directory.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
feat(idf.py): Allow adding arguments from file via @filename.txt (#11783) (GitHub PR)
Closes IDFGH-10584 and IDFGH-10539
See merge request espressif/esp-idf!24955
The ticket below reports some very strange behaviour, where
even a simple command like the following
python3 -c "import subprocess; subprocess.run([''])"
actually spawns some process and pass instead of raising
PermissionError. Even though the problem is most probably somewhere
else, not in idf_tools, we may just return early if there is no
command available for get_version().
Closes https://github.com/espressif/esp-idf/issues/11880
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When interrupt allocation fails, esp_intr_alloc will now print a
message telling that no free interrupt was found. This message is
then checked in hints.yml, to give a link to the troubleshooting
guide.
Currently ESP_PYTHON is used in the install.sh script, but it's
overwritten in detect_python.sh. This allows to explicitly specify
python binary, which should be used to create venv.
$ ESP_PYTHON=/usr/bin/python3.7 ./install.{sh,fish}
$ ESP_PYTHON=/usr/bin/python3.7 ./export.{sh,fish}
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Now the soc interrupts definitions are scattered around in the esp-idf
which are out of sync. Put interrupts definitions in soc/periph_defs.h
(!ESP32) or soc/soc.h(ESP32) together in soc/interrupts.h.
idf.py spawns gdb process within a thread and uses Thread.join() to wait
for the gdb process to finish. As CTRL+C(SIGINT) is used by gdb to interrupt the
running program, we catch the SIGINT while waiting on the gdb to finish,
and try Thread.join() again.
With cpython's commit
commit a22be4943c119fecf5433d999227ff78fc2e5741
Author: Victor Stinner <vstinner@python.org>
Date: Mon Sep 27 14:20:31 2021 +0200
bpo-45274: Fix Thread._wait_for_tstate_lock() race condition (GH-28532)
this logic doesn't work anymore, because cpython internally marks the
thread as stopped when join() is interrupted with an exception. IMHO
this is broken in cpython and there is a bug report about this
https://github.com/python/cpython/issues/90882. Problem is that
waiting on a thread to finish is based on acquiring a lock. Meaning
join() is waiting on _tstate_lock. If this wait is interrupted, the
above referenced commit adds a logic that checks if the lock is help,
meaning the thread is done and marks the thread as stopped. But there is
no way to tell if the lock was acquired by us running join() or if it's
held by someone else e.g. still by the thread bootstrap code. Meaning
the thread is still running.
I may be missing something, but I don't see any reason why to spawn gdb
process within a thread. This change removes the thread and spawns gdb
directly. Instead waiting on a thread, we wait on the process to finish,
replacing join() with wait() and avoiding this problem.
Closes https://github.com/espressif/esp-idf/issues/11871
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Usage of `flushregs` in gdb will result in the following warning:
```
Warning: 'flushregs', an alias for the command 'maintenance flush register-cache', is deprecated.
Use 'maintenance flush register-cache'.
```
This patch updates to use the recommend command.
Signed-off-by: Daniel Mangum <georgedanielmangum@gmail.com>
Merges https://github.com/espressif/esp-idf/pull/11535
There are some register description errors in parts of rom/gpio.h
This commit update the incorrect comments in rom/gpio.h And now
esp_rom_gpio.h is recommend instead of rom/gpio.h. So this commit adds
macro SIG_GPIO_OUT_IDX in esp_rom_gpio_connect_out_signal
function and removes the reference to gpio.h in some source files.
Closes https://github.com/espressif/esp-idf/issues/11737
Verify that curses tool can be successfully imported on unix systems
When detected:
- installing esp-idf -> reinstall python environment
- using idf.py menuconfig -> raise error with hint message
Closes https://github.com/espressif/esp-idf/issues/11643
fix(hal/include): fix header violations in hal component
fix(hal/include): Move type definitions from `xx_hal.h` to `xx_types.h`
fix(hal/include): Move type definitions from `xx_hal.h` to `xx_types.h`
fix(hal/include): Add comment for a far away `#endif`
fix(hal/include): change scope for cpp guard
ci: Remove components/hal/ comment from public headers check exceptions
Add missing include macro sdkconfig.h for header files
Add missing include macro stdbool.h for header files
Add missing include macro stdint.h for header files
Add missing capability guard macro for header files
Add missing cpp guard macro for header files
Remove some useless include macros
Add some missing `inline` attribute for functions defined in header files
Remove components/hal/ from public headers check exceptions
fix(hal/include): fix invalid licenses
fix(hal/include): fix invalid licenses
fix(hal/include): add missing soc_caps.h
fix(hal): include soc_caps.h before cap macro is used
fix(hal): Remove unnecessary target check
fix(hal): fix header and macro problems
Add missing include macro
Remove loop dependency in hal
Add comment for far-away endif
fix(hal): Add missing soc_caps.h
ci: update check_copyright_ignore.txt
Change the sequence of `#include` macro, cpp guard macro
Change the wrap scope of capacity macro
fix(hal): Change position of C++ guard to pass test
- add hardware stack guard based on assist-debug module
- enable hardware stack guard by default
- disable hardware stack guard for freertos ci.release test
- refactor rtos_int_enter/rtos_int_exit to change SP register inside them
- fix panic_reason.h header for RISC-V
- update docs to include information about the new feature
Current version of the test is using "git-submodule foreach", which
requires submodules to be initialized. Non-initialized submodules are
ignored. Our CI is not performing submodule initialization, but instead
it only downloads the submodule content in tools/ci/ci_fetch_submodule.py
from cache and copies it into the submodule path.
Since we already know the submodule path from .gitconfig, we can use it
as argument to git-ls-tree and avoid calling git-submodule at all. This
allows to perform the test even if the submodules are not initialization
and also it makes the code simpler.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This uses the test from CI introduced in MR !23989. Even though the
original CI test is run with pytest, it doesn't use any pytest specific
code/features and it should never need them. So it make sense just to
re-use the code.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The script works the same way with httpd and softap mode, but it's a bit
confusing to provision the device over Ethernet or USB and call the
transport "softap". That's why we introduce an alias called httpd which
uses the same idea (http server with service name), but that service
runs on any interface (where the specified service is available)
As pointed out by Fu Hanxi, the pytest_build_system job is currently
using --parallel-index and --parallel-count, which are provided by
pytest-embedded, so we should not disable it. Moreover to properly
disable pytest-embedded we should use "no:pytest_embedded". Meaning
this probably was not working as indented anyway.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This adds SBOM information for submodules, which are not managed
by Espressif. Meaning there is no fork for them in the espressif
namespace. Other submodules should add sbom.yml manifest file to
the root of their git repository.
The SBOM information for submodules is stored in the .gitmodules file.
Each SBOM related variable has the "sbom-" prefix and the following
variables may be used:
sbom-version:
submodule version
sbom-cpe:
CPE record if available in NVD. This will be used by the SBOM
tool to check for possible submodule vulnerabilities. The
version in the CPE can be replaced with the "{}" placeholder,
which will be replaced by the "sbom-version" value from above.
sbom-supplier:
Person or organization who is providing the submodule.
It has to start with "Person:" or "Organization:" prefix
as required by the SPDX-2.2 standard.
sbom-url:
URL to the project if exists, e.g. github.
sbom-description:
Project description.
sbom-hash:
Submodule SHA as recorded in the git-tree. This field is used by
CI to check that the submodule checkout hash and info in .gitmodules
are in sync. IOW if submodule is updated and it has SBOM info in
.gitmodules, the .gitmodules has to be updated too. The test is
part of this commit. The checkout has of the submodule can be found
by using "git submodule status".
Example for micro-ecc submodule
---8<---
[submodule "components/bootloader/subproject/components/micro-ecc/micro-ecc"]
path = components/bootloader/subproject/components/micro-ecc/micro-ecc
url = ../../kmackay/micro-ecc.git
sbom-version = 1.0
sbom-cpe = cpe:2.3🅰️micro-ecc_project:micro-ecc:{}:*:*:*:*:*:*:*
sbom-supplier = Person: Ken MacKay
sbom-url = https://github.com/kmackay/micro-ecc
sbom-description = A small and fast ECDH and ECDSA implementation for 8-bit, 32-bit, and 64-bit processors
sbom-hash = d037ec89546fad14b5c4d5456c2e23a71e554966
---8<---
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This extends information provided in the project_description.json file.
Newly added information can be used in the SBOM generating tool and
also to improve hints regarding the the component dependency issues.
Added fields
version:
This adds versioning to the project_description.json file,
so it's easy to identify if it contains the required information.
project_version:
Can be used as a version for the resulting binary e.g. `hello_world.bin`.
idf_path:
This one is probably not necessary, but it allows tools to run even without
esp-idf environment exported(e.g. export.sh).
c_compiler:
The `CMAKE_C_COMPILER` value with full path to the compiler binary. This can
be used to get information about toolchain, which was used to build the project.
common_component_reqs:
List of common components as presented in cmake's __COMPONENT_REQUIRES_COMMON
and set in tools/cmake/build.cmake:__build_init().
build_component_info:
Detailed information about components used during build. It's a
dictionary with the component name as a key and each component has
a dictionary with detailed information. Following is an example for
the efuse component.
"efuse": {
"alias": "idf::efuse",
"target": "___idf_efuse",
"prefix": "idf",
"dir": "/home/fhrbata/work/esp-idf/components/efuse",
"type": "LIBRARY",
"lib": "__idf_efuse",
"reqs": [],
"priv_reqs": [ "bootloader_support", "soc", "spi_flash" ],
"managed_reqs": [],
"managed_priv_reqs": [],
"file": "/home/fhrbata/work/blink/build/esp-idf/efuse/libefuse.a",
"sources": [ "/home/fhrbata/work/esp-idf/components/efuse/esp32s3/esp_efuse_table.c", ... ],
"include_dirs": [ "include", "esp32s3/include" ]
}
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently make_json_list() returns '[ "" ]' for empty cmake list. Fix this
so empty json list is returned instead.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
- This features allows the NVS encryption keys to be derived and protected using
the HMAC peripheral. Since the encryption keys are derived at runtime, they
are not stored anywhere in the flash and hence this feature does not require
a separate `nvs_keys` partition.
This commit adds support for handling multiple requests simultaneously by introducing two new functions: `httpd_req_async_handler_begin()` and `httpd_req_async_handler_complete()`. These functions allow creating an asynchronous copy of a request that can be used on a separate thread and marking the asynchronous request as completed, respectively.
Additionally, a new flag `for_async_req` has been added to the `httpd_sess_t` struct to indicate if a socket is being used for an asynchronous request and should not be purged from the LRU cache.
An example have been added to demonstrate the usage of these new functions.
Closes https://github.com/espressif/esp-idf/issues/10594
Signed-off-by: Harshit Malpani <harshit.malpani@espressif.com>
esp_bootloader_format: Adds bootloader description structure to read bootloader version from app
Closes IDFGH-7206 and IDFGH-7573
See merge request espressif/esp-idf!21592
spi_flash: One more step for supporting flash suspend. 1. Support more esp chips 2. Improve real-time performance 3. Make timing more stable
See merge request espressif/esp-idf!22755
Currently hints are processed only once the process is finished and
exits with non-error exit code. In interactive mode, e.g. for monitor,
we want to process ouput lines for hints right away.
This adds a simple buffer, which keeps the last line and once EOL is
reached, it is processed for hints.
Since the original hints processing was file based, a new helper
function was added to allow processing hints in string.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
* reverted old faulty mechanism to set different
standard according to the toolchain
* Using -std=gnu++2b now for both gcc and clang
* Added a build test app to check the C++ standard in IDF
* Updated english docs to reflect the change to C++23
Roland Dobai