From 1d8eece8348fa2f45f9a396ad75a6d9590b6e98e Mon Sep 17 00:00:00 2001 From: Angus Gratton Date: Thu, 1 Sep 2016 09:58:47 +1000 Subject: [PATCH] Rename README.buildenv to docs/build_system.rst and ReST-ify it --- README.buildenv | 265 ------------------------------------ docs/build_system.rst | 302 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 302 insertions(+), 265 deletions(-) delete mode 100644 README.buildenv create mode 100644 docs/build_system.rst diff --git a/README.buildenv b/README.buildenv deleted file mode 100644 index 0fc007e0ca..0000000000 --- a/README.buildenv +++ /dev/null @@ -1,265 +0,0 @@ -The build structure of the Espressif IoT Development Framework explained. - -An ESP-IDF project can be seen as an almagation of a number of components. -For example, for a webserver that shows the current humidity, we would -have: -- The ESP32 base libraries (libc, rom bindings etc) -- The WiFi drivers -- A TCP/IP stack -- The FreeRTOS operating system -- A webserver -- A driver for an humidity sensor -- Main code tying it all together - -ESP-IDF makes these components explicit and configurable. To do that, when a project -is compiled, the build environment will look up all the components in the -ESP-IDF directories, the project directories and optionally custom other component -directories. It then allows the user to configure compile-time options using -a friendly text-based menu system to customize the ESP-IDF as well as other components -to the requirements of the project. After the components are customized, the -build process will compile everything into an output file, which can then be uploaded -into a board in a way that can also be defined by components. - -A project in this sense is defined as a directory under which all the files required -to build it live, excluding the ESP-IDF files and the toolchain. A simple project -tree looks like this: - -- myProject/ - build/ - - components/ - component1/ - Makefile - - Kconfig - - src1.c - - component2/ - Makefile - - Kconfig - - src1.c - - main/ - src1.c - - src2.c - - Makefile - - -As we can see, a project consists of a components/ subdirectory containing its -components as well as one or more directories containing the project-specific -sources; by default a single directory called 'main' is assumed. The project -directory will also have a Makefile where the projects name as well as optionally -other options are defined. After compilation, the project directory will contain -a 'build'-directory containing all of the objects, libraries and other generated -files as well as the final binary. - -Components also have a Makefile containing various definititions influencing -the build process of the component as well as the project it's used in, as -well as a Kconfig file defining the compile-time options that are settable -by means of the menu system. - - -Project makefile variables that can be set by the programmer: -PROJECT_NAME: Mandatory. Name for the project -BUILD_DIR_BASE: Set the directory where all objects/libraries/binaries end up in. - Defaults to $(PROJECT_PATH)/build -COMPONENT_DIRS: Search path for components. Defaults to the component/ directories - in the ESP-IDF path and the project path. -COMPONENTS: A list of component names. Defaults to all the component found in the - COMPONENT_DIRS directory -EXTRA_COMPONENT_DIRS: Defaults to unset. Use this to add directories to the default - COMPONENT_DIRS. -SRCDIRS: Directories under the project dir containing project-specific sources. - Defaults to 'main'. These are treated as 'lite' components: they do not have - include directories that are passed to the compilation pass of all components and - they do not have a Kconfig option. - - -Component makefile variables that can be set by the programmer: -COMPONENT_ADD_INCLUDEDIRS: Relative path to include directories to be added to - the entire project -COMPONENT_PRIV_INCLUDEDIRS: Relative path to include directories that are only used - when compiling this specific component -COMPONENT_DEPENDS: Names of any components that need to be compiled before this component. -COMPONENT_ADD_LDFLAGS: Ld flags to add for this project. Defaults to -l$(COMPONENT_NAME). - Add libraries etc in the current directory as $(abspath libwhatever.a) -COMPONENT_EXTRA_INCLUDES: Any extra include paths. These will be prefixed with '-I' and - passed to the compiler; please put absolute paths here. -COMPONENT_SRCDIRS: Relative directories to look in for sources. Defaults to '.', the current - directory (the root of the component) only. Use this to specify any subdirectories. Note - that specifying this overwrites the default action of compiling everything in the - components root dir; to keep this behaviour please also add '.' as a directory in this - list. -COMPONENT_OBJS: Object files to compile. Defaults to the .o variants of all .c and .S files - that are found in COMPONENT_SRCDIRS. -COMPONENT_EXTRA_CLEAN: Files that are generated using rules in the components Makefile - that also need to be cleaned -COMPONENT_BUILDRECIPE: Recipe to build the component. Optional. Defaults to building all - COMPONENT_OBJS and linking them into lib(componentname).a -COMPONENT_CLEANRECIPE: Recipe to clean the component. Optional. Defaults to removing - all built objects and libraries. -COMPONENT_BUILD_DIR: Equals the cwd of the component build, which is the build dir - of the component (where all the .o etc files should be created). - - -These variables are already set early on in the Makefile and the values in it will -be usable in component or project Makefiles: -CC, LD, AR, OBJCOPY: Xtensa gcc tools -HOSTCC, HOSTLD etc: Host gcc tools -LDFLAGS, CFLAGS: Set to usable values as defined in ESP-IDF Makefile -PROJECT_NAME: Name of the project, as set in project makefile -PROJECT_PATH: Path to the root of the project folder -COMPONENTS: Name of the components to be included -CONFIG_*: Values set by 'make menuconfig' also have corresponding Makefile variables. - -For components, there also are these defines: -COMPONENT_PATH: Absolute path to the root of the source tree of the component we're - compiling -COMPONENT_LIBRARY: The full path to the static library the components compilation pass - is supposed to generate - - -How this works: -The Make process is always invoked from the project directory by the -user; invoking it anywhere else gives an error. This is what happens if -we build a binary: - -The Makefile first determines how it was included. It determines it was -included as a project file in this case and will continue to figure out -various paths as well as the components available to it. It will also -collect the ldflags and includes that the components specify they need. -It does this by running a dummy Make on the components with a target that -will output these values. - -The Makefile will then create targets to build the lib*.a libraries of -all components and make the elf target depend on this. The targets -invoke Make on the makefiles of the components in a subshell: this way -the components have full freedom to do whatever is necessary to build -the library without influencing other components. By default, the -component includes the utility makefile $(IDF_PATH)/make/component_common.mk. -This provides default targets and configurations that will work -out-of-the-box for most projects. - -For components that have parts that need to be run when building of the -project is done, you can create a file called Makefile.projbuild in the -component root directory. This file will be included in the main -Makefile. For the menu, there's an equivalent: if you want to include -options not in the 'components' submenu, create a Kconfig.projbuild and -it will be included in the main menu of menuconfig. Take good care when -(re)defining stuff here: because it's included with all the other -.projbuild files, it's possible to overwrite variables or re-declare -targets defined in the ESP-IDF makefile/Kconfig and other .projbuild files - - -WRITING COMPONENT MAKEFILES - -A component consists of a directory which doubles as the name for the -component: a component named 'httpd' lives in a directory called 'httpd' -Because components usually live under the project directory (although -they can also reside in an other folder), the path to this may be -something like /home/myuser/projects/myprojects/components/httpd . - -One of the things that most components will have is a Makefile, -containing instructions on how to build the component. Because the -build environment tries to set reasonable defaults that will work most -of the time, a component Makefile can be pretty small. At the absolute -minimum, it will just include the ESP-IDF component makefile, which adds -component functionality: - -----8<---- -include $(IDF_PATH)/make/component_common.mk ----->8---- - -This will take all the .c and .S files in the component root and compile -them into object files, finally linking them into a library. -Subdirectories are ignored; if your project has sources in subdirectories -instead of in the root of the component, you can tell that to the build -system by setting COMPONENT_SRCDIRS: - -----8<---- -COMPONENT_SRCDIRS := src1 src2 -include $(IDF_PATH)/make/component_common.mk ----->8---- - -This will compile all source files in the src1/ and src2/ subdirectories -instead. - -The standard component.mk logic adds all .S and .c files in the source -directories as sources to be compiled unconditionally. It is possible -to circumvent that logic and hardcode the objects to be compiled by -manually setting the COMPONENT_OBJS variable to the name of the -objects that need to be generated: - -----8<---- -COMPONENT_OBJS := file1.o file2.o thing/filea.o thing/fileb.o anotherthing/main.o -include $(IDF_PATH)/make/component_common.mk ----->8---- - -This can also be used in order to conditionally compile some files -dependent on the options selected in the Makefile: - -Kconfig: -----8<---- -config FOO_ENABLE_BAR - bool "Enable the BAR feature." - help - This enables the BAR feature of the FOO component. ----->8---- - -Makefile: -----8<---- -COMPONENT_OBJS := foo_a.o foo_b.o $(if $(CONFIG_FOO_ENABLE_BAR),foo_bar.o foo_bar_interface.o) -include $(IDF_PATH)/make/component_common.mk ----->8---- - -Some components will have a situation where a source file isn't supplied -with the component itself but has to be generated from another file. Say -our component has a header file that consists of the converted binary -data of a BMP file, converted using a hypothetical tool called bmp2h. The -header file is then included in as C source file called graphics_lib.c. - -----8<---- -COMPONENT_EXTRA_CLEAN := logo.h - -graphics_lib.o: logo.h - -logo.h: $(COMPONENT_PATH)/logo.bmp - bmp2h -i $^ -o $@ - -include $(IDF_PATH)/make/component_common.mk ----->8---- - -In this example, graphics_lib.o and logo.h will be generated in the -current directory (the build directory) while logo.bmp comes with the -component and resides under the component path. Because logo.h is a -generated file, it needs to be cleaned when make clean is called which -why it is added to the COMPONENT_EXTRA_CLEAN variable. - - -This will work just fine, but there's one last cosmetic improvement that -can be done. The make system tries to make the make process somewhat -easier on the eyes by hiding the commands (unless you run make with the -V=1 switch) and this does not do that yet. Here's an improved version -that will output in the same style as the rest of the make process: - -----8<---- -COMPONENT_EXTRA_CLEAN := test_tjpgd_logo.h - -graphics_lib.o: logo.h - -logo.h: $(COMPONENT_PATH)/logo.bmp - $(summary) BMP2H $@ - $(Q) bmp2h -i $^ -o $@ - -include $(IDF_PATH)/make/component_common.mk ----->8---- - -Obviously, there are cases where all these recipes are insufficient for a -certain component, for example when the component is basically a wrapper -around another third-party component not originally intended to be -compiled under this build system. In that case, it's possible to forego -the build system entirely by setting COMPONENT_OWNBUILDTARGET and -possibly COMPONENT_OWNCLEANTARGET and defining your own build- and clean -target. The build target can do anything as long as it creates -$(COMPONENT_LIBRARY) for the main file to link into the project binary, -and even that is not necessary: if the COMPONENT_ADD_LDFLAGS variable -is set, the component can instruct the linker to do anything else as well. - - - - - - - - diff --git a/docs/build_system.rst b/docs/build_system.rst new file mode 100644 index 0000000000..24381019b5 --- /dev/null +++ b/docs/build_system.rst @@ -0,0 +1,302 @@ +Build System +------------ + +This document explains the Espressif IoT Development Framework build system and the +concept of "components" + +Read this document if you want to know how to organise a new ESP-IDF project. + +We recommend using the esp-idf-template_ project as a starting point for your project. + +Overview +======== + +An ESP-IDF project can be seen as an almagation of a number of components. +For example, for a webserver that shows the current humidity, we would +have: + +- The ESP32 base libraries (libc, rom bindings etc) +- The WiFi drivers +- A TCP/IP stack +- The FreeRTOS operating system +- A webserver +- A driver for an humidity sensor +- Main code tying it all together + +ESP-IDF makes these components explicit and configurable. To do that, when a project +is compiled, the build environment will look up all the components in the +ESP-IDF directories, the project directories and optionally custom other component +directories. It then allows the user to configure compile-time options using +a friendly text-based menu system to customize the ESP-IDF as well as other components +to the requirements of the project. After the components are customized, the +build process will compile everything into an output file, which can then be uploaded +into a board in a way that can also be defined by components. + +A project in this sense is defined as a directory under which all the files required +to build it live, excluding the ESP-IDF files and the toolchain. A simple project +tree might look like this:: + + - myProject/ - build/ + - components/ - component1/ - component.mk + - Kconfig + - src1.c + - component2/ - component.mk + - Kconfig + - src1.c + - main/ - src1.c + - src2.c + - Makefile + +As we can see, a project consists of a components/ subdirectory containing its +components as well as one or more directories containing the project-specific +sources; by default a single directory called 'main' is assumed. The project +directory will also have a Makefile where the projects name as well as optionally +other options are defined. After compilation, the project directory will contain +a 'build'-directory containing all of the objects, libraries and other generated +files as well as the final binary. + +Components also have a custom makefile - ``component.mk``. This contains various definititions +influencing the build process of the component as well as the project it's used +in. Components may also include a Kconfig file defining the compile-time options that are +settable by means of the menu system. + +Project makefile variables that can be set by the programmer:: + + PROJECT_NAME: Mandatory. Name for the project + BUILD_DIR_BASE: Set the directory where all objects/libraries/binaries end up in. + Defaults to $(PROJECT_PATH)/build + COMPONENT_DIRS: Search path for components. Defaults to the component/ directories + in the ESP-IDF path and the project path. + COMPONENTS: A list of component names. Defaults to all the component found in the + COMPONENT_DIRS directory + EXTRA_COMPONENT_DIRS: Defaults to unset. Use this to add directories to the default + COMPONENT_DIRS. + SRCDIRS: Directories under the project dir containing project-specific sources. + Defaults to 'main'. These are treated as 'lite' components: they do not have + include directories that are passed to the compilation pass of all components and + they do not have a Kconfig option. + +Component makefile variables that can be set by the programmer:: + + COMPONENT_ADD_INCLUDEDIRS: Relative path to include directories to be added to + the entire project + COMPONENT_PRIV_INCLUDEDIRS: Relative path to include directories that are only used + when compiling this specific component + COMPONENT_DEPENDS: Names of any components that need to be compiled before this component. + COMPONENT_ADD_LDFLAGS: Ld flags to add for this project. Defaults to -l$(COMPONENT_NAME). + Add libraries etc in the current directory as $(abspath libwhatever.a) + COMPONENT_EXTRA_INCLUDES: Any extra include paths. These will be prefixed with '-I' and + passed to the compiler; please put absolute paths here. + COMPONENT_SRCDIRS: Relative directories to look in for sources. Defaults to '.', the current + directory (the root of the component) only. Use this to specify any subdirectories. Note + that specifying this overwrites the default action of compiling everything in the + components root dir; to keep this behaviour please also add '.' as a directory in this + list. + COMPONENT_OBJS: Object files to compile. Defaults to the .o variants of all .c and .S files + that are found in COMPONENT_SRCDIRS. + COMPONENT_EXTRA_CLEAN: Files that are generated using rules in the components Makefile + that also need to be cleaned + COMPONENT_BUILDRECIPE: Recipe to build the component. Optional. Defaults to building all + COMPONENT_OBJS and linking them into lib(componentname).a + COMPONENT_CLEANRECIPE: Recipe to clean the component. Optional. Defaults to removing + all built objects and libraries. + COMPONENT_BUILD_DIR: Equals the cwd of the component build, which is the build dir + of the component (where all the .o etc files should be created). + +These variables are already set early on in the Makefile and the values in it will +be usable in component or project Makefiles:: + + CC, LD, AR, OBJCOPY: Xtensa gcc tools + HOSTCC, HOSTLD etc: Host gcc tools + LDFLAGS, CFLAGS: Set to usable values as defined in ESP-IDF Makefile + PROJECT_NAME: Name of the project, as set in project makefile + PROJECT_PATH: Path to the root of the project folder + COMPONENTS: Name of the components to be included + CONFIG_*: All values set by 'make menuconfig' have corresponding Makefile variables. + +For components, there also are these defines:: + + COMPONENT_PATH: Absolute path to the root of the source tree of the component we're + compiling + COMPONENT_LIBRARY: The full path to the static library the components compilation pass + is supposed to generate + +Make Process +------------ + +The Make process is always invoked from the project directory by the +user; invoking it anywhere else gives an error. This is what happens if +we build a binary: + +The Makefile first determines how it was included. It figures out +various paths as well as the components available to it. It will also +collect the ldflags and includes that the components specify they need. +It does this by running a dummy make on the components with a "get_variable" +target that will output these values. + +The Makefile will then create targets to build the lib*.a libraries of +all components and make the elf target depend on this. The main Makefile +invokes Make on the componen.mk of each components inside a sub-mke: this way +the components have full freedom to do whatever is necessary to build +the library without influencing other components. By default, the +component.mk includes the utility makefile $(IDF_PATH)/make/component_common.mk. +This provides default targets and configurations that will work +out-of-the-box for most projects. + +KConfig +------- + +Each component can also have a Kconfig file, alongside the component.mk, that contains +details to add to "menuconfig" for this component. + +Makefile.projbuild +------------------ + +For components that have parts that need to be run when building of the +project is done, you can create a file called Makefile.projbuild in the +component root directory. This file will be included in the main +Makefile. + + +KConfig.projbuild +----------------- + +There's an equivalent to Makefile.projbuild for KConfig: if you want to include +options at the top-level, not inside the 'components' submenu then create a Kconfig.projbuild and +it will be included in the main menu of menuconfig. + +Take good care when (re)defining stuff here: because it's included with all the other +.projbuild files, it's possible to overwrite variables or re-declare targets defined in +the ESP-IDF makefile/Kconfig and other .projbuild files. It's generally better to just +create a KConfig file, if you can. + + +Writing Component Makefiles +--------------------------- + +A component consists of a directory which doubles as the name for the +component: a component named 'httpd' lives in a directory called 'httpd' +Because components usually live under the project directory (although +they can also reside in an other folder), the path to this may be +something like /home/myuser/projects/myprojects/components/httpd . + +One of the things that most components will have is a component.mk makefile, +containing instructions on how to build the component. Because the +build environment tries to set reasonable defaults that will work most +of the time, component.mk can be very small. + +Simplest component.mk +===================== + +At the minimum, component.mk will just include the ESP-IDF component "common" makefile, +which adds common component functionality:: + + include $(IDF_PATH)/make/component_common.mk + +This will take all the .c and .S files in the component root and compile +them into object files, finally linking them into a library. + + +Adding source directories +========================= + +By default, subdirectories are ignored. If your project has sources in subdirectories +instead of in the root of the component then you can tell that to the build +system by setting COMPONENT_SRCDIRS:: + + COMPONENT_SRCDIRS := src1 src2 + include $(IDF_PATH)/make/component_common.mk + +This will compile all source files in the src1/ and src2/ subdirectories +instead. + +Specifying source files +======================= + +The standard component.mk logic adds all .S and .c files in the source +directories as sources to be compiled unconditionally. It is possible +to circumvent that logic and hardcode the objects to be compiled by +manually setting the COMPONENT_OBJS variable to the name of the +objects that need to be generated:: + + COMPONENT_OBJS := file1.o file2.o thing/filea.o thing/fileb.o anotherthing/main.o + include $(IDF_PATH)/make/component_common.mk + + +Adding conditional configuration +================================ + +The configuration system can be used to conditionally compile some files +dependending on the options selected in ``make menuconfig``: + +Kconfig:: + config FOO_ENABLE_BAR + bool "Enable the BAR feature." + help + This enables the BAR feature of the FOO component. + +Makefile:: + COMPONENT_OBJS := foo_a.o foo_b.o $(if $(CONFIG_FOO_ENABLE_BAR),foo_bar.o foo_bar_interface.o) + include $(IDF_PATH)/make/component_common.mk + + +Source Code Generation +====================== + +Some components will have a situation where a source file isn't supplied +with the component itself but has to be generated from another file. Say +our component has a header file that consists of the converted binary +data of a BMP file, converted using a hypothetical tool called bmp2h. The +header file is then included in as C source file called graphics_lib.c:: + + COMPONENT_EXTRA_CLEAN := logo.h + + graphics_lib.o: logo.h + + logo.h: $(COMPONENT_PATH)/logo.bmp + bmp2h -i $^ -o $@ + + include $(IDF_PATH)/make/component_common.mk + +In this example, graphics_lib.o and logo.h will be generated in the +current directory (the build directory) while logo.bmp comes with the +component and resides under the component path. Because logo.h is a +generated file, it needs to be cleaned when make clean is called which +why it is added to the COMPONENT_EXTRA_CLEAN variable. + +Cosmetic Improvements +===================== + +The above example will work just fine, but there's one last cosmetic +improvement that can be done. The make system tries to make the make +process somewhat easier on the eyes by hiding the commands (unless you +run make with the V=1 switch) and this does not do that yet. Here's an +improved version that will output in the same style as the rest of the +make process:: + + COMPONENT_EXTRA_CLEAN := test_tjpgd_logo.h + + graphics_lib.o: logo.h + + logo.h: $(COMPONENT_PATH)/logo.bmp + $(summary) BMP2H $@ + $(Q) bmp2h -i $^ -o $@ + + include $(IDF_PATH)/make/component_common.mk + +Fully Overriding The Component Makefile +--------------------------------------- + +Obviously, there are cases where all these recipes are insufficient for a +certain component, for example when the component is basically a wrapper +around another third-party component not originally intended to be +compiled under this build system. In that case, it's possible to forego +the build system entirely by setting COMPONENT_OWNBUILDTARGET and +possibly COMPONENT_OWNCLEANTARGET and defining your own build- and clean +target. The build target can do anything as long as it creates +$(COMPONENT_LIBRARY) for the main file to link into the project binary, +and even that is not necessary: if the COMPONENT_ADD_LDFLAGS variable +is set, the component can instruct the linker to do anything else as well. + + +.. _esp-idf-template: https://github.com/espressif/esp-idf-template