Merge branch 'feat/tools_python_constraints_env' into 'master'

Tools: Disable Python constraint files with environment variable

Closes IDFGH-7722

See merge request espressif/esp-idf!18830

docs/en/api-guides/tools/idf-tools.rst

@@ -109,9 +109,9 @@ Any mirror server can be used provided the URL matches the ``github.com`` downlo
 
 * ``check``: For each tool, checks whether the tool is available in the system path and in ``IDF_TOOLS_PATH``.
 
-* ``install-python-env``: Create a Python virtual environment in the ``${IDF_TOOLS_PATH}/python_env`` directory and install there the required Python packages. An optional ``--features`` argument allows one to specify a comma-separated list of features to be added or removed. Feature that begins with ``-`` will be removed and features with ``+`` or without any sign will be added. Example syntax for removing feature ``XY`` is ``--features=-XY`` and for adding ``--features=+XY`` or ``--features=XY``. If both removing and adding options are provided with the same feature, no operation is performed. For each feature a requirements file must exist. For example, feature ``XY`` is a valid feature if ``${IDF_PATH}/tools/requirements/requirements.XY.txt`` is an existing file with a list of Python packages to be installed. There is one mandatory ``core`` feature ensuring core functionality of ESP-IDF (build, flash, monitor, debug in console). There can be an arbitrary number of optional features. The selected list of features is stored in ``idf-env.json``. The requirement files contain a list of the desired Python packages to be installed and ``espidf.constraints.*.txt`` downloaded from https://dl.espressif.com and stored in ``${IDF_TOOLS_PATH}`` the package version requirements for a given ESP-IDF version.
+* ``install-python-env``: Create a Python virtual environment in the ``${IDF_TOOLS_PATH}/python_env`` directory and install there the required Python packages. An optional ``--features`` argument allows one to specify a comma-separated list of features to be added or removed. Feature that begins with ``-`` will be removed and features with ``+`` or without any sign will be added. Example syntax for removing feature ``XY`` is ``--features=-XY`` and for adding ``--features=+XY`` or ``--features=XY``. If both removing and adding options are provided with the same feature, no operation is performed. For each feature a requirements file must exist. For example, feature ``XY`` is a valid feature if ``${IDF_PATH}/tools/requirements/requirements.XY.txt`` is an existing file with a list of Python packages to be installed. There is one mandatory ``core`` feature ensuring core functionality of ESP-IDF (build, flash, monitor, debug in console). There can be an arbitrary number of optional features. The selected list of features is stored in ``idf-env.json``. The requirement files contain a list of the desired Python packages to be installed and ``espidf.constraints.*.txt`` downloaded from https://dl.espressif.com and stored in ``${IDF_TOOLS_PATH}`` the package version requirements for a given ESP-IDF version. Althought it is not recommended, the download and use of constraint files can be disabled with the ``--no-constraints`` argument or setting the ``IDF_PYTHON_CHECK_CONSTRAINTS`` environment variable to ``no``.
 
-* ``check-python-dependencies``: Checks if all required Python packages are installed. Packages from ``${IDF_PATH}/tools/requirements/requirements.*.txt`` files selected by the feature list of ``idf-env.json`` are checked with the package versions specified in the ``espidf.constraints.*.txt`` file. The constraint file will be downloaded from https://dl.espressif.com if this step hasn't been done already in the last day.
+* ``check-python-dependencies``: Checks if all required Python packages are installed. Packages from ``${IDF_PATH}/tools/requirements/requirements.*.txt`` files selected by the feature list of ``idf-env.json`` are checked with the package versions specified in the ``espidf.constraints.*.txt`` file. The constraint file will be downloaded from https://dl.espressif.com if this step hasn't been done already in the last day. The use of constraints files can be disabled similarly to the ``install-python-env`` command.
 
 * ``uninstall``: Print and remove tools, that are currently not used by active ESP-IDF version.
 

tools/docker/Dockerfile

@@ -75,6 +75,10 @@ RUN : \
   && rm -rf $IDF_TOOLS_PATH/dist \
   && :
 
+# The constraint file has been downloaded and the right Python package versions installed. No need to check and
+# download this at every invocation of the container.
+ENV IDF_PYTHON_CHECK_CONSTRAINTS=no
+
 # Ccache is installed, enable it by default
 ENV IDF_CCACHE_ENABLE=1
 COPY entrypoint.sh /opt/esp/entrypoint.sh

tools/idf_tools.py

@@ -1879,8 +1879,7 @@ def get_constraints(idf_version):  # type: (str) -> str
     try:
         age = datetime.date.today() - datetime.date.fromtimestamp(os.path.getmtime(constraint_path))
         if age < datetime.timedelta(days=1):
-            info(f'Skipping the download of {constraint_path} because it was downloaded recently. If you believe '
+            info(f'Skipping the download of {constraint_path} because it was downloaded recently.')
-                 f'that this is causing you trouble then remove it manually and re-run your install script.')
             return constraint_path
     except OSError:
         # doesn't exist or inaccessible
@@ -1902,6 +1901,7 @@ def get_constraints(idf_version):  # type: (str) -> str
         return constraint_path
     else:
         fatal('Failed to download, and retry count has expired')
+        info('See the help on how to disable constraints in order to work around this issue.')
         raise DownloadError()
 
 
@@ -2402,6 +2402,8 @@ def main(argv):  # type: (list[str]) -> None
     uninstall.add_argument('--dry-run', help='Print unused tools.', action='store_true')
     uninstall.add_argument('--remove-archives', help='Remove old archive versions and archives from unused tools.', action='store_true')
 
+    no_constraints_default = os.environ.get('IDF_PYTHON_CHECK_CONSTRAINTS', '').lower() in ['0', 'n', 'no']
+
     if IDF_MAINTAINER:
         for subparser in [download, install]:
             subparser.add_argument('--mirror-prefix-map', nargs='*',
@@ -2419,9 +2421,10 @@ def main(argv):  # type: (list[str]) -> None
     install_python_env.add_argument('--no-index', help='Work offline without retrieving wheels index')
     install_python_env.add_argument('--features', default='core', help='A comma separated list of desired features for installing.'
                                                                        ' It defaults to installing just the core funtionality.')
-    install_python_env.add_argument('--no-constraints', action='store_true', default=False,
+    install_python_env.add_argument('--no-constraints', action='store_true', default=no_constraints_default,
                                     help='Disable constraint settings. Use with care and only when you want to manage '
-                                         'package versions by yourself.')
+                                         'package versions by yourself. It can be set with the IDF_PYTHON_CHECK_CONSTRAINTS '
+                                         'environment variable.')
 
     if IDF_MAINTAINER:
         add_version = subparsers.add_parser('add-version', help='Add or update download info for a version')
@@ -2446,9 +2449,10 @@ def main(argv):  # type: (list[str]) -> None
 
     check_python_dependencies = subparsers.add_parser('check-python-dependencies',
                                                       help='Check that all required Python packages are installed.')
-    check_python_dependencies.add_argument('--no-constraints', action='store_true', default=False,
+    check_python_dependencies.add_argument('--no-constraints', action='store_true', default=no_constraints_default,
                                            help='Disable constraint settings. Use with care and only when you want '
-                                                'to manage package versions by yourself.')
+                                                'to manage package versions by yourself. It can be set with the IDF_PYTHON_CHECK_CONSTRAINTS '
+                                                'environment variable.')
 
     args = parser.parse_args(argv)