micropython/ports/unix
Maarten van der Schrieck 3bca93b2d0 ports: Fix sys.stdout.buffer.write() return value.
MicroPython code may rely on the return value of sys.stdout.buffer.write()
to reflect the number of bytes actually written. While in most scenarios a
write() operation is successful, there are cases where it fails, leading to
data loss. This problem arises because, currently, write() merely returns
the number of bytes it was supposed to write, without indication of
failure.

One scenario where write() might fail, is where USB is used and the
receiving end doesn't read quickly enough to empty the receive buffer. In
that case, write() on the MicroPython side can timeout, resulting in the
loss of data without any indication, a behavior observed notably in
communication between a Pi Pico as a client and a Linux host using the ACM
driver.

A complex issue arises with mp_hal_stdout_tx_strn() when it involves
multiple outputs, such as USB, dupterm and hardware UART. The challenge is
in handling cases where writing to one output is successful, but another
fails, either fully or partially. This patch implements the following
solution:

mp_hal_stdout_tx_strn() attempts to write len bytes to all of the possible
destinations for that data, and returns the minimum successful write
length.

The implementation of this is complicated by several factors:
- multiple outputs may be enabled or disabled at compiled time
- multiple outputs may be enabled or disabled at runtime
- mp_os_dupterm_tx_strn() is one such output, optionally containing
  multiple additional outputs
- each of these outputs may or may not be able to report success
- each of these outputs may or may not be able to report partial writes

As a result, there's no single strategy that fits all ports, necessitating
unique logic for each instance of mp_hal_stdout_tx_strn().

Note that addressing sys.stdout.write() is more complex due to its data
modification process ("cooked" output), and it remains unchanged in this
patch. Developers who are concerned about accurate return values from
write operations should use sys.stdout.buffer.write().

This patch might disrupt some existing code, but it's also expected to
resolve issues, considering that the peculiar return value behavior of
sys.stdout.buffer.write() is not well-documented and likely not widely
known. Therefore, it's improbable that much existing code relies on the
previous behavior.

Signed-off-by: Maarten van der Schrieck <maarten@thingsconnected.nl>
2023-12-22 10:32:46 +11:00
..
mbedtls extmod/mbedtls: Enable certificate time/date validation by default. 2023-12-01 15:08:11 +11:00
variants extmod/modmachine: Add MICROPY_PY_MACHINE_PIN_BASE option. 2023-11-30 16:11:11 +11:00
Makefile extmod/modmachine: Factor ports' machine module dict to common code. 2023-11-30 16:11:11 +11:00
README.md py/mkrules.mk: Don't strip binary if STRIP variable is unset. 2023-09-29 13:01:12 +10:00
alloc.c unix/alloc: Use MP_REGISTER_ROOT_POINTER() for mmap_region_head. 2022-07-18 13:49:09 +10:00
coverage.c unix: Update port to use the new event functions. 2023-12-08 13:17:15 +11:00
coveragecpp.cpp esp32,unix: Support building C++ code. 2020-10-29 15:29:50 +11:00
fatfs_port.c unix/fatfs_port: Fix month offset in timestamp calculation. 2020-09-02 00:19:06 +10:00
gccollect.c py/gc: Make gc_dump_info/gc_dump_alloc_table take a printer as argument. 2023-03-10 10:58:10 +11:00
input.c all: Update to point to files in new shared/ directory. 2021-07-12 17:08:10 +10:00
input.h ports: Make new ports/ sub-directory and move all ports there. 2017-09-06 13:40:51 +10:00
main.c py/builtinevex: Handle invalid filenames for execfile. 2023-10-12 15:17:59 +11:00
modffi.c unix/modffi: Move header includes inside MICROPY_PY_FFI guard. 2022-10-14 14:36:17 +11:00
modjni.c unix/modjni: Fix build errors with type definitions and error strings. 2023-09-04 10:22:19 +10:00
modmachine.c extmod/modmachine: Add MICROPY_PY_MACHINE_PIN_BASE option. 2023-11-30 16:11:11 +11:00
modos.c all: Rename *umodule*.c to remove the "u" prefix. 2023-06-08 17:54:17 +10:00
modsocket.c unix/modsocket: Add poll support for missing ERR,HUP,NVAL poll values. 2023-08-06 11:54:06 +10:00
modtermios.c all: Fix spelling mistakes based on codespell check. 2023-04-27 18:03:06 +10:00
modtime.c all: Rename *umodule*.c to remove the "u" prefix. 2023-06-08 17:54:17 +10:00
mpbthciport.c extmod/modbluetooth: Make all HCI transports trace in the same format. 2023-04-26 11:37:20 +10:00
mpbtstackport.h unix: Implement BLE H4 HCI UART for btstack/nimble. 2020-09-08 12:53:24 +10:00
mpbtstackport_common.c extmod/btstack: Switch to use hci_dump_init instead of hci_dump_open. 2023-04-22 00:39:31 +10:00
mpbtstackport_h4.c extmod/btstack: Update BTstack bindings to work with latest BTstack. 2022-10-22 14:28:25 +11:00
mpbtstackport_usb.c extmod/btstack: Update BTstack bindings to work with latest BTstack. 2022-10-22 14:28:25 +11:00
mpconfigport.h unix: Update port to use the new event functions. 2023-12-08 13:17:15 +11:00
mpconfigport.mk all: Rename UMODULE to MODULE in preprocessor/Makefile vars. 2023-06-08 17:54:11 +10:00
mphalport.h ports: Move definitions of ATOMIC_SECTION macros to mphalport.h. 2023-12-01 14:37:48 +11:00
mpnimbleport.c extmod/nimble: Make stm32 and unix NimBLE ports use synchronous events. 2020-11-13 17:19:05 +11:00
mpnimbleport.h unix: Implement BLE H4 HCI UART for btstack/nimble. 2020-09-08 12:53:24 +10:00
mpthreadport.c py/modthread: Return thread id from start_new_thread(). 2023-09-03 18:49:18 +10:00
mpthreadport.h unix: Implement `-X realtime` command-line option on macOS. 2022-05-24 00:51:47 +10:00
qstrdefsport.h all: Add *FORMAT-OFF* in various places. 2020-02-28 10:31:07 +11:00
unix_mphal.c ports: Fix sys.stdout.buffer.write() return value. 2023-12-22 10:32:46 +11:00

README.md

The Unix version

The "unix" port requires a standard Unix-like environment with gcc and GNU make. This includes Linux, BSD, macOS, and Windows Subsystem for Linux. The x86 and x64 architectures are supported (i.e. x86 32- and 64-bit), as well as ARM and MIPS. Making a full-featured port to another architecture requires writing some assembly code for the exception handling and garbage collection. Alternatively, a fallback implementation based on setjmp/longjmp can be used.

To build (see section below for required dependencies):

$ cd ports/unix
$ make submodules
$ make

Then to give it a try:

$ ./build-standard/micropython
>>> list(5 * x + y for x in range(10) for y in [4, 2, 1])

Use CTRL-D (i.e. EOF) to exit the shell.

Learn about command-line options (in particular, how to increase heap size which may be needed for larger applications):

$ ./build-standard/micropython -h

To run the complete testsuite, use:

$ make test

The Unix port comes with a built-in package manager called mip, e.g.:

$ ./build-standard/micropython -m mip install hmac

or

$ ./build-standard/micropython
>>> import mip
>>> mip.install("hmac")

Browse available modules at micropython-lib. See Package management for more information about mip.

External dependencies

The libffi library and pkg-config tool are required. On Debian/Ubuntu/Mint derivative Linux distros, install build-essential(includes toolchain and make), libffi-dev, and pkg-config packages.

Other dependencies can be built together with MicroPython. This may be required to enable extra features or capabilities, and in recent versions of MicroPython, these may be enabled by default. To build these additional dependencies, in the unix port directory first execute:

$ make submodules

This will fetch all the relevant git submodules (sub repositories) that the port needs. Use the same command to get the latest versions of submodules as they are updated from time to time. After that execute:

$ make deplibs

This will build all available dependencies (regardless whether they are used or not). If you intend to build MicroPython with additional options (like cross-compiling), the same set of options should be passed to make deplibs. To actually enable/disable use of dependencies, edit the ports/unix/mpconfigport.mk file, which has inline descriptions of the options. For example, to build the SSL module, MICROPY_PY_SSL should be set to 1.

Debug Symbols

By default, builds are stripped of symbols and debug information to save size.

To build a debuggable version of the Unix port, there are two options

  1. Run make [other arguments] DEBUG=1. Note setting DEBUG also reduces the optimisation level, so it's not a good option for builds that also want the best performance.
  2. Run make [other arguments] STRIP=. Note that the value of STRIP is empty. This will skip the build step that strips symbols and debug information, but changes nothing else in the build configuration.