This commit adds a tool, codeformat.py, which will reformat C and Python
code to fit a certain style. By default the tool will reformat (almost)
all the original (ie not 3rd-party) .c, .h and .py files in this
repository. Passing filenames on the command-line to codeformat.py will
reformat only those. Reformatting is done in-place.
uncrustify is used for C reformatting, which is available for many
platforms and can be easily built from source, see
https://github.com/uncrustify/uncrustify. The configuration for uncrustify
is also added in this commit and values are chosen to best match the
existing code style. A small post-processing stage on .c and .h files is
done by codeformat.py (after running uncrustify) to fix up some minor
items:
- space inserted after * when used as multiplication with sizeof
- #if/ifdef/ifndef/elif/else/endif are dedented by one level when they are
configuring if-blocks and case-blocks.
For Python code, the formatter used is black, which can be pip-installed;
see https://github.com/psf/black. The defaults are used, except for line-
length which is set at 99 characters to match the "about 100" line-length
limit used in C code.
The formatting tools used and their configuration were chosen to strike a
balance between keeping existing style and not changing too many lines of
code, and enforcing a relatively strict style (especially for Python code).
This should help to keep the code consistent across everything, and reduce
cognitive load when writing new code to match the style.
This option makes pyboard.py exit as soon as the script/command is
successfully sent to the device, ie it does not wait for any output. This
can help to avoid hangs if the board is being rebooted with --comman (for
example).
Example usage:
$ python3 ./tools/pyboard.py --device /dev/ttyUSB0 --no-follow \
--command 'import machine; machine.reset()'
Some parts of code have been aligned to increase readability. In general
'' instead of "" were used wherever possible to keep the same convention
for entire file. Import inspect line has been moved to the top according
to hints reported by pep8 tools. A few extra spaces were removed, a few
missing spaces were added. Comments have been updated, mostly in
"read_dfu_file" function. Some other comments have been capitalized and/or
slightly updated. A few docstrings were fixed as well. No real code
changes intended.
This is an alternative to f4ed2df that adds a newline so that the output of
the test starts on a new line and the result of the test is prefixed with
"result: " to distinguish it from the test output.
Suggested-by: @dpgeorge
This patch allows executing .mpy files (including native ones) directly on
a target, eg a board over a serial connection. So there's no need to copy
the file to its filesystem to test it.
For example:
$ mpy-cross foo.py
$ pyboard.py foo.mpy
This makes the loading of viper-code-with-relocations a bit neater and
easier to understand, by treating the rodata/bss like a special object to
be loaded into the constant table (which is how it behaves).
We don't want to add a feature flag to .mpy files that indicate float
support because it will get complex and difficult to use. Instead the .mpy
is built using whatever precision it chooses (float or double) and the
native glue API will convert between this choice and what the host runtime
actually uses.
This commit adds a new tool called mpy_ld.py which is essentially a linker
that builds .mpy files directly from .o files. A new header file
(dynruntime.h) and makefile fragment (dynruntime.mk) are also included
which allow building .mpy files from C source code. Such .mpy files can
then be dynamically imported as though they were a normal Python module,
even though they are implemented in C.
Converting .o files directly (rather than pre-linked .elf files) allows the
resulting .mpy to be more efficient because it has more control over the
relocations; for example it can skip PLT indirection. Doing it this way
also allows supporting more architectures, such as Xtensa which has
specific needs for position-independent code and the GOT.
The tool supports targets of x86, x86-64, ARM Thumb and Xtensa (windowed
and non-windowed). BSS, text and rodata sections are supported, with
relocations to all internal sections and symbols, as well as relocations to
some external symbols (defined by dynruntime.h), and linking of qstrs.
Usage:
mpy-tool.py -o merged.mpy --merge mod1.mpy mod2.mpy
The constituent .mpy files are executed sequentially when the merged file
is imported, and they all use the same global namespace.
While the new manifest.py style got introduced for freezing python code
into the resulting binary, the old way - where files and modules within
ports/*/modules where baked into the resulting binary - was still
supported via `freeze('$(PORT_DIR)/modules')` within manifest.py.
However behaviour changed for symlinked directories (=modules), as those
links weren't followed anymore.
This commit restores the original behaviour by explicitly following
symlinks within a modules/ directory
When loading a manifest file, e.g. by include(), it will chdir first to the
directory of that manifest. This means that all file operations within a
manifest are relative to that manifest's location.
As a consequence of this, additional environment variables are needed to
find absolute paths, so the following are added: $(MPY_LIB_DIR),
$(PORT_DIR), $(BOARD_DIR). And rename $(MPY) to $(MPY_DIR) to be
consistent.
Existing manifests are updated to match.
This introduces a new build variable FROZEN_MANIFEST which can be set to a
manifest listing (written in Python) that describes the set of files to be
frozen in to the firmware.
Instead of encoding 4 zero bytes as placeholders for the simple_name and
source_file qstrs, and storing the qstrs after the bytecode, store the
qstrs at the location of these 4 bytes. This saves 4 bytes per bytecode
function stored in a .mpy file (for example lcd160cr.mpy drops by 232
bytes, 4x 58 functions). And resulting code size is slightly reduced on
ports that use this feature.
This patch compresses the second part of the bytecode prelude which
contains the source file name, function name, source-line-number mapping
and cell closure information. This part of the prelude now begins with a
single varible length unsigned integer which encodes 2 numbers, being the
byte-size of the following 2 sections in the header: the "source info
section" and the "closure section". After decoding this variable unsigned
integer it's possible to skip over one or both of these sections very
easily.
This scheme saves about 2 bytes for most functions compared to the original
format: one in the case that there are no closure cells, and one because
padding was eliminated.
The start of the bytecode prelude contains 6 numbers telling the amount of
stack needed for the Python values and exceptions, and the signature of the
function. Prior to this patch these numbers were all encoded one after the
other (2x variable unsigned integers, then 4x bytes), but using so many
bytes is unnecessary.
An entropy analysis of around 150,000 bytecode functions from the CPython
standard library showed that the optimal Shannon coding would need about
7.1 bits on average to encode these 6 numbers, compared to the existing 48
bits.
This patch attempts to get close to this optimal value by packing the 6
numbers into a single, varible-length unsigned integer via bit-wise
interleaving. The interleaving scheme is chosen to minimise the average
number of bytes needed, and at the same time keep the scheme simple enough
so it can be implemented without too much overhead in code size or speed.
The scheme requires about 10.5 bits on average to store the 6 numbers.
As a result most functions which originally took 6 bytes to encode these 6
numbers now need only 1 byte (in 80% of cases).
Prior to this patch mp_opcode_format would calculate the incorrect size of
the MP_BC_UNWIND_JUMP opcode, missing the additional byte. But, because
opcodes below 0x10 are unused and treated as bytes in the .mpy load/save
and freezing code, this bug did not show any symptoms, since nested unwind
jumps would rarely (if ever) reach a depth of 16 (so the extra byte of this
opcode would be between 0x01 and 0x0f and be correctly loaded/saved/frozen
simply as an undefined opcode).
This patch fixes this bug by correctly accounting for the additional byte.
.
- Split 'qemu-arm' from 'unix' for generating tests.
- Add frozen module to the qemu-arm test build.
- Add test that reproduces the requirement to half-word align native
function data.
Use "-f" to select filesystem mode, followed by the command to execute.
Optionally put ":" at the start of a filename to indicate that it's on the
remote device, if it would otherwise be ambiguous.
Examples:
$ pyboard.py -f ls
$ pyboard.py -f cat main.py
$ pyboard.py -f cp :main.py . # get from device
$ pyboard.py -f cp main.py : # put to device
$ pyboard.py -f rm main.py
Previously, when linking qstr objects in native code for ARM Thumb, the
index into the machine code was being incremented by 4, not 8. It should
be 8 to account for the size of the two machine instructions movw and movt.
This patch makes sure the index into the machine code is incremented by the
correct amount for all variations of qstr linking.
See issue #4829.
Fixes errors in the tool when 1) linking qstrs in native ARM-M code; 2)
freezing multiple files some of which use native code and some which don't.
Fixes issue #4829.
The user can now select their own package index by either passing the "-i"
command line option, or setting the upip.index_urls variable (before doing
an install).
The https://micropython.org/pi package index hosts packages from
micropython-lib and will be searched first when installing a package. If a
package is not found here then it will fallback to PyPI.
Prior to this patch, when a lot of data was output by a running script
pyboard.py would try to capture all of this output into the "data"
variable, which would gradually slow down pyboard.py to the point where it
would have large CPU and memory usage (on the host) and potentially lose
data.
This patch fixes this problem by not accumulating the data in the case that
the data is not needed, which is when "data_consumer" is used.
The qstr window size is not log-2 encoded, it's just the actual number (but
in mpy-tool.py this didn't lead to an error because the size is just used
to truncate the window so it doesn't grow arbitrarily large in memory).
Addresses issue #4635.
This system makes it a lot easier to include external libraries as static,
native modules in MicroPython. Simply pass USER_C_MODULES (like
FROZEN_MPY_DIR) as a make parameter.
Damien George