Although the original motivation given for the workaround[1] is correct,
nlr.o and nlrthumb.o are linked with a small enough distance that the
problem does not occur, and the workaround isn't necessary. The distance
between the b instruction and its target (nlr_push_tail) is just 64
bytes[2], well within the ±2046 byte range addressable by an
unconditional branch instruction in Thumb mode.
The workaround induces a relocation in the text section (textrel), which
isn't supported everywhere, notably not on musl-libc[3], where it causes
a crash on start-up. With the workaround removed, micropython works on an
ARMv5T Linux system built with musl-libc.
This commit changes nlrthumb.c to use a direct jump by default, but
leaves the long jump workaround as an option for those cases where it's
actually needed.
[1]: commit dd376a239d
Author: Damien George <damien.p.george@gmail.com>
Date: Fri Sep 1 15:25:29 2017 +1000
py/nlrthumb: Get working again on standard Thumb arch (ie not Thumb2).
"b" on Thumb might not be long enough for the jump to nlr_push_tail so
it must be done indirectly.
[2]: Excerpt from objdump -d micropython:
000095c4 <nlr_push_tail>:
95c4: b510 push {r4, lr}
95c6: 0004 movs r4, r0
95c8: f02d fd42 bl 37050 <mp_thread_get_state>
95cc: 6943 ldr r3, [r0, #20]
95ce: 6023 str r3, [r4, #0]
95d0: 6144 str r4, [r0, #20]
95d2: 2000 movs r0, #0
95d4: bd10 pop {r4, pc}
000095d6 <nlr_pop>:
95d6: b510 push {r4, lr}
95d8: f02d fd3a bl 37050 <mp_thread_get_state>
95dc: 6943 ldr r3, [r0, #20]
95de: 681b ldr r3, [r3, #0]
95e0: 6143 str r3, [r0, #20]
95e2: bd10 pop {r4, pc}
000095e4 <nlr_push>:
95e4: 60c4 str r4, [r0, #12]
95e6: 6105 str r5, [r0, #16]
95e8: 6146 str r6, [r0, #20]
95ea: 6187 str r7, [r0, #24]
95ec: 4641 mov r1, r8
95ee: 61c1 str r1, [r0, #28]
95f0: 4649 mov r1, r9
95f2: 6201 str r1, [r0, #32]
95f4: 4651 mov r1, sl
95f6: 6241 str r1, [r0, #36] @ 0x24
95f8: 4659 mov r1, fp
95fa: 6281 str r1, [r0, #40] @ 0x28
95fc: 4669 mov r1, sp
95fe: 62c1 str r1, [r0, #44] @ 0x2c
9600: 4671 mov r1, lr
9602: 6081 str r1, [r0, #8]
9604: e7de b.n 95c4 <nlr_push_tail>
[3]: https://www.openwall.com/lists/musl/2020/09/25/4
Signed-off-by: J. Neuschäfer <j.ne@posteo.net>
Enabled by MICROPY_COMPILE_ALLOW_TOP_LEVEL_AWAIT. When enabled, this means
that scope such as module-level functions and REPL statements can yield.
The outer C code must then handle this yielded generator.
Signed-off-by: Damien George <damien@micropython.org>
This commit implements the 'e' half-float format: 10-bit mantissa, 5-bit
exponent. It uses native _Float16 if supported by the compiler, otherwise
uses custom bitshifting encoding/decoding routines.
Signed-off-by: Matthias Urlichs <matthias@urlichs.de>
Signed-off-by: Damien George <damien@micropython.org>
Add `pop()`, `appendleft()`, and `extend()` methods, support iteration
and indexing, and initializing from an existing sequence.
Iteration and indexing (subscription) have independent configuration flags
to enable them. They are enabled by default at the same level that
collections.deque is enabled (the extra features level).
Also add tests for checking new behavior.
Signed-off-by: Damien George <damien@micropython.org>
Disabled by default, but enabled on all boards that previously had
`MICROPY_PY_MACHINE_BARE_METAL_FUNCS` enabled.
Signed-off-by: Damien George <damien@micropython.org>
The STATIC macro was introduced a very long time ago in commit
d5df6cd44a. The original reason for this was
to have the option to define it to nothing so that all static functions
become global functions and therefore visible to certain debug tools, so
one could do function size comparison and other things.
This STATIC feature is rarely (if ever) used. And with the use of LTO and
heavy inline optimisation, analysing the size of individual functions when
they are not static is not a good representation of the size of code when
fully optimised.
So the macro does not have much use and it's simpler to just remove it.
Then you know exactly what it's doing. For example, newcomers don't have
to learn what the STATIC macro is and why it exists. Reading the code is
also less "loud" with a lowercase static.
One other minor point in favour of removing it, is that it stops bugs with
`STATIC inline`, which should always be `static inline`.
Methodology for this commit was:
1) git ls-files | egrep '\.[ch]$' | \
xargs sed -Ei "s/(^| )STATIC($| )/\1static\2/"
2) Do some manual cleanup in the diff by searching for the word STATIC in
comments and changing those back.
3) "git-grep STATIC docs/", manually fixed those cases.
4) "rg -t python STATIC", manually fixed codegen lines that used STATIC.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
For boards with MICROPY_CONFIG_ROM_LEVEL_AT_LEAST_CORE_FEATURES and up.
This gets samd21 boards working (which need the vfs module in _boot.py),
B_L072Z_LRWAN1, and nrf boards with smaller MCUs that use CORE or BASIC
feature levels.
Signed-off-by: robert-hh <robert@hammelrath.com>
This will apply to bare-arm and minimal, as well as the minimal unix
variant.
Change the default to MICROPY_QSTR_BYTES_IN_HASH=1 for the CORE,BASIC
levels, 2 for >=EXTRA.
Removes explicit setting of MICROPY_QSTR_BYTES_IN_HASH==1 in ports that
don't set the feature level (because 1 is implied by the default level,
CORE). Applies to cc3200, pic16bt, powerpc.
Removes explicit setting for nRF (which sets feature level). Also for samd,
which sets CORE for d21 and FULL for d51. This means that d21 is unchanged
with MICROPY_QSTR_BYTES_IN_HASH==1, but d51 now moves from 1 to 2 (roughly
adds 1kiB).
The only remaining port which explicitly set bytes-in-hash is rp2 because
it's high-flash (hence CORE level) but lowish-SRAM, so it's worthwhile
saving the RAM for runtime qstrs.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
MICROPY_BEGIN_ATOMIC_SECTION/MICROPY_END_ATOMIC_SECTION belong more to the
MicroPython HAL rather than build configuration settings, so move their
default configuration to py/mphal.h, and require all users of these macros
to include py/mphal.h (here, py/objexcept.c and py/scheduler.c).
This helps ports separate configuration from their HAL implementations, and
can improve build times (because mpconfig.h is included everywhere, whereas
mphal.h is not).
Signed-off-by: Damien George <damien@micropython.org>
This provides a way to enable features and changes slated for MicroPython
2.x, by running `make MICROPY_PREVIEW_VERSION_2=1`. Also supported for
the cmake ports (except Zephyr).
This is an alternative to having a 2.x development branch (or equivalently,
keeping a 1.x release branch). Any feature or change that needs to be
"hidden" until 2.x can use this flag (either in the Makefile or the
preprocessor).
A good example is changing function arguments or other public API features,
in particular to aid in improving consistency between ports.
When `MICROPY_PREVIEW_VERSION_2` is enabled, the REPL banner is amended to
say "MicroPython (with v2.0 preview) vX.Y.Z", and sys.implementation gets a
new field `_v2` set to `True`.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
See https://github.com/micropython/micropython/issues/12127 for details.
Previously at the point when a release is made, we update mpconfig.h
and set a git tag. i.e. the version increments at the release.
Now the version increments immediately after the release. The workflow is:
1. Final commit in the cycle updates mpconfig.h to set (X, Y, 0, 0) (i.e.
clear the pre-release state).
2. This commit is tagged "vX.Y.0".
3. First commit for the new cycle updates mpconfig.h to set (X, Y+1, 0, 1)
(i.e. increment the minor version, set the pre-release state).
4. This commit is tagged "vX.Y+1.0-preview".
The idea is that a nightly build is actually a "preview" of the _next_
release. i.e. any documentation describing the current release may not
actually match the nightly build. So we use "preview" as our semver
pre-release identifier.
Changes in this commit:
- Add MICROPY_VERSION_PRERELEASE to mpconfig.h to allow indicating that
this is not a release version.
- Remove unused MICROPY_VERSION integer.
- Append "-preview" to MICROPY_VERSION_STRING when the pre-release state
is set.
- Update py/makeversionhdr.py to no longer generate MICROPY_GIT_HASH.
- Remove the one place MICROPY_GIT_HASH was used (it can use
MICROPY_GIT_TAG instead).
- Update py/makeversionhdr.py to also understand
MICROPY_VERSION_PRERELEASE in mpconfig.h.
- Update py/makeversionhdr.py to convert the git-describe output into
semver-compatible "X.Y.Z-preview.N.gHASH".
- Update autobuild.sh to generate filenames using the new scheme.
- Update remove_old_firmware.py to match new scheme.
- Update mpremote's pyproject.toml to handle the "-preview" suffix in the
tag. setuptools_scm maps to this "rc0" to match PEP440.
- Fix docs heading where it incorrectly said "vvX.Y.Z" for release docs.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The rp2 port was enabling SSL and had finalizers enabled via the "extra
features" level, but missed explicitly enabling `MICROPY_PY_SSL_FINALISER`
(like esp32, stm32, and mimxrt did).
This commit makes `MICROPY_PY_SSL_FINALISER` default to enabled if
finalizers are enabled, and removes the explicit setting of this for
esp32, stm32, mimxrt (because they all use the "extra features" level).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The only reason that const had to be disabled was to make the test output
match CPython when const was involved. Instead, this commit fixes the test
to handle the lines where const is used.
Also:
- remove the special handling for MICROPY_PERSISTENT_CODE_SAVE in
unix/mpconfigport.h, and make this automatic.
- move the check for MICROPY_PERSISTENT_CODE_SAVE to where it's used (like
we do for other similar checks) and add a comment explaining it.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When set, the split heap is automatically extended with new areas on
demand, and shrunk if a heap area becomes empty during a GC pass or soft
reset.
To save code size the size allocation for a new heap block (including
metadata) is estimated at 103% of the failed allocation, rather than
working from the more complex algorithm in gc_try_add_heap(). This appears
to work well except in the extreme limit case when almost all RAM is
exhausted (~last few hundred bytes). However in this case some allocation
is likely to fail soon anyhow.
Currently there is no API to manually add a block of a given size to the
heap, although that could easily be added if necessary.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
A previous commit removed the unix-specific select module implementation
and made unix use the common one.
This commit adds an optimisation so that the system poll function is used
when polling objects that have a file descriptor. With this optimisation
enabled, if code registers both file-descriptor-based objects, and non-
file-descriptor-based objects with select.poll() then the following occurs:
- the system poll is called for all file-descriptor-based objects with a
timeout of 1ms
- then the bare-metal polling implementation is used for remaining objects,
which calls into their ioctl method (which can be in C or Python)
In the case where all objects have file descriptors, the system poll is
called with the full timeout requested by the caller. That makes it as
efficient as possible in the case everything has a file descriptor.
Benefits of this approach:
- all ports use the same select module implementation
- the unix port now supports polling of all objects and matches bare metal
implementations
- it's still efficient for existing cases where only files and sockets are
polled (on unix)
- the bare metal implementation does not change
- polling of SSL objects will now work on unix by calling in to the ioctl
method on SSL objects (this is required for asyncio ssl support)
Note that extmod/vfs_posix_file.c has poll disable when the optimisation is
enabled, because the code is not reachable when the optimisation is used.
Signed-off-by: Damien George <damien@micropython.org>
Previously this was explicitly enabled on esp32/stm32/renesas/mimxrt/samd,
but didn't get a default feature level because it wasn't in py/mpconfig.h.
With this commit it's now enabled at the "extra features" level, which adds
rp2, unix-standard, windows, esp8266, webassembly, and some nrf boards.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This provides similar functionality to the former zlib.DecompIO and
especially CPython's gzip.GzipFile for both compression and decompression.
This class can be used directly, and also can be used from Python to
implement (via io.BytesIO) zlib.decompress and zlib.compress, as well as
gzip.GzipFile.
Enable/disable this on all ports/boards that zlib was previously configured
for.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This will be replaced with a new deflate module providing the same
functionality, with an optional frozen Python wrapper providing a
replacement zlib module.
binascii.crc32 is temporarily disabled.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The asyncio module now has much better CPython compatibility and
deserves to be just called "asyncio".
This will avoid people having to write `from uasyncio import asyncio`.
Renames all files, and updates port manifests to use the new path. Also
renames the built-in _uasyncio to _asyncio.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is a MicroPython-specific module that existed to support the old
version of uasyncio. It's undocumented and not enabled on all ports and
takes up code size unnecessarily.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Previously sys.path could be modified by append/pop or slice assignment.
This allows `sys.path = [...]`, which can be simpler in many cases, but
also improves CPython compatibility.
It also allows sys.path to be set to a tuple which means that you can
clear sys.path (e.g. temporarily) with no allocations.
This also makes sys.path (and sys.argv for consistency) able to be disabled
via mpconfig. The unix port (and upytesthelper) require them, so they
explicitly verify that they're enabled.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Otherwise you can get into the confusing state where e.g. sys.ps1 is
enabled in config (via `MICROPY_PY_SYS_PS1_PS2`) but still doesn't actually
get enabled.
Also verify that the required delegation options are enabled in modsys.c.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This replaces the previous QSTR_null entry in the globals dict which could
leak out to Python (e.g. via iteration of mod.__dict__) and could lead to
crashes.
It results in smaller code size at the expense of turning a lookup into a
loop, but the list it is looping over likely only contains one or two
elements.
To allow a module to register its custom attr function it can use the new
`MP_REGISTER_MODULE_DELEGATION` macro.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
In order to keep "import umodule" working, the existing mechanism is
replaced with a simple fallback to drop the "u".
This makes importing of built-ins no longer touch the filesystem, which
makes a typical built-in import take ~0.15ms rather than 3-5ms.
(Weak links were added in c14a81662c)
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This can lead to duplicate initialisations if a module can be imported
via multiple names, so the module must track this itself anyway.
This reduces code size (diff is -40 bytes), and avoids special treatment of
builtin-modules-with-init with respect to sys.modules. No other builtin
modules get put into sys.modules.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
To use this:
- Create a built-in module, and add the module object as a member of the
parent module's globals dict.
- The submodule can set its `__name__` to either `QSTR_foo_dot_bar` or
`QSTR_bar`. The former requires using qstrdefs(port).h to make the qstr.
Because `bar` is a member of `foo`'s globals, it is possible to write
`import foo` and then immediately use `foo.bar` without importing it
explicitly. This means that if `bar` has an `__init__`, it will not be
called in this situation, and for that reason, sub-modules should not have
`__init__` methods. If this is required, then all initalisation for
sub-modules should be done by the top-level module's (i.e. `foo`'s)
`__init__` method.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Changes in this commit:
- Add MICROPY_GC_HOOK_LOOP to gc_info() and gc_alloc(). Both of these can
be long running (many milliseconds) which is too long to be blocking in
some applications.
- Pass loop variable to MICROPY_GC_HOOK_LOOP(i) macro so that implementers
can use it, e.g. to improve performance by only calling a function every
X number of iterations.
- Drop outer call to MICROPY_GC_HOOK_LOOP in gc_mark_subtree().
Based on extmod/utime_mphal.c, with:
- a globals dict added
- time.localtime wrapper added
- time.time wrapper added
- time.time_ns function added
New configuration options are added for this module:
- MICROPY_PY_UTIME (enabled at basic features level)
- MICROPY_PY_UTIME_GMTIME_LOCALTIME_MKTIME
- MICROPY_PY_UTIME_TIME_TIME_NS
Signed-off-by: Damien George <damien@micropython.org>
This is intended to be used by the very outer caller of the VM/runtime. It
allows setting a top-level NLR handler that can be jumped to directly, in
order to forcefully abort the VM/runtime.
Enable using:
#define MICROPY_ENABLE_VM_ABORT (1)
Set up the handler at the top level using:
nlr_buf_t nlr;
nlr.ret_val = NULL;
if (nlr_push(&nlr) == 0) {
nlr_set_abort(&nlr);
// call into the VM/runtime
...
nlr_pop();
} else {
if (nlr.ret_val == NULL) {
// handle abort
...
} else {
// handle other exception that propagated to the top level
...
}
}
nlr_set_abort(NULL);
Schedule an abort, eg from an interrupt handler, using:
mp_sched_vm_abort();
Signed-off-by: Damien George <damien@micropython.org>
The C-level printf is usually used for internal debugging prints, and a
port/board may want to redirect this somewhere other than stdout.
Signed-off-by: Damien George <damien@micropython.org>
When you want to use the valgrind memory analysis tool on MicroPython, you
can arrange to define MICROPY_DEBUG_VALGRIND to enable use of special
valgrind macros. For now, this only fixes `gc_get_ptr` so that it never
emits the diagnostic "Conditional jump or move depends on uninitialised
value(s)".
Signed-off-by: Jeff Epler <jepler@gmail.com>
This module is useful, but it is not always needed. Disabling it saves
several kilobytes of build size, depending on other config options.
Signed-off-by: Laurens Valk <laurens@pybricks.com>
Only intended to be used on Unix and other "OS" ports. Matches CPython.
This should give the absolute path to the executing binary.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
The existings mp_obj_type_t uses a sparse representation for slots for the
capability methods of the type (eg print, make_new). This commit adds a
compact slot-index representation. The basic idea is that where the
mp_obj_type_t struct used to have 12 pointer fields, it now has 12 uint8_t
indices, and a variable-length array of pointers. So in the best case (no
fields used) it saves 12x4-12=36 bytes (on a 32-bit machine) and in the
common case (three fields used) it saves 9x4-12=24 bytes.
Overall with all associated changes, this slot-index representation reduces
code size by 1000 to 3000 bytes on bare-metal ports. Performance is
marginally better on a few tests (eg about 1% better on misc_pystone.py and
misc_raytrace.py on PYBv1.1), but overall marginally worse by a percent or
so.
See issue #7542 for further analysis and discussion.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
J. Neuschäfer