From 2adc54c4176c068634eae6c0d210d7210a4cdf65 Mon Sep 17 00:00:00 2001 From: Peter Hinch Date: Sat, 7 Sep 2019 11:55:12 +0100 Subject: [PATCH] V0.28 release. --- FONT_TO_PY.md | 60 ++++++++++++++++++++++++++++++++--------------- font_test.py | 17 ++++++++++---- font_to_py.py | 37 ++++++++++++++++++++--------- writer/DRIVERS.md | 35 +++++++++++++++++---------- 4 files changed, 101 insertions(+), 48 deletions(-) diff --git a/FONT_TO_PY.md b/FONT_TO_PY.md index 5f9fda0..0d74b56 100644 --- a/FONT_TO_PY.md +++ b/FONT_TO_PY.md @@ -5,12 +5,21 @@ is to save RAM on resource-limited targets: the font file may be incorporated into a firmware build such that it occupies flash memory rather than scarce RAM. Python code built into firmware is known as frozen bytecode. +## V0.27/0.28 notes + +7 Sept 2019 + +Remove redundancy from index file: significantly reduces file size for sparse +fonts. Add a comment field in the output file showing creation command line. +Repo includes the file `extended`. This facilitates creating fonts with useful +scientific glyphs. Improvements to `font_test.py`. + ###### [Main README](./README.md) # Dependencies The utility requires Python 3.2 or greater, also `freetype` which may be -installed using `pip3`. On Linux at a root prompt: +installed using `pip3`. On Linux (you may need a root prompt): ```shell # apt-get install python3-pip @@ -25,10 +34,11 @@ required height in pixels and outputs a Python 3 source file. The pixel layout is determined by command arguments. By default fonts are stored in variable pitch form. This may be overidden by a command line argument. -By default the ASCII character set (ordinal values 32 to 126 inclusive) is -supported. Command line arguments can modify this range as required, if -necessary to include extended ASCII characters up to 255. Alternatively non -English or non-contiguous character sets may be defined. +By default the printable ASCII character set (ordinal values 32 to 126 +inclusive) is supported (i.e. not including control characters). Command line +arguments can modify this range as required to specify arbitrary sets of +Unicode characters. Non-English and non-contiguous character sets may be +defined. Further arguments ensure that the byte contents and layout are correct for the target display hardware. Their usage should be specified in the documentation @@ -55,7 +65,7 @@ Example usage to produce a file `myfont.py` with height of 23 pixels: 32 (ASCII space). * -l or --largest Ordinal value of largest character to be stored. Default 126. * -e or --errchar Ordinal value of character to be rendered if an attempt is - made to display an out-of-range character. Default 63 (ASCII "?"). + made to display an out-of-range character. Default 63 (ord("?")). * -i or --iterate Specialist use. See below. * -c or --charset Option to restrict the characters in the font to a specific set. See below. @@ -63,8 +73,8 @@ Example usage to produce a file `myfont.py` with height of 23 pixels: for alternative character sets such as Cyrillic: the file must contain the character set to be included. An example file is `cyrillic`. Another is `extended` which adds unicode characters "° μ π ω ϕ θ α β γ δ λ Ω" to those - with `ord` values from 32-128. Such files will only produce useful results if - the font file includes them. + with `ord` values from 32-126. Such files will only produce useful results if + the source font file includes those glyphs. The -c option may be used to reduce the size of the font file by limiting the character set. If the font file is frozen as bytecode this will not reduce RAM @@ -76,22 +86,21 @@ $ font_to_py.py Arial.ttf 20 arial_clock.py -c 1234567890: Example usage with the -k option: ```shell font_to_py.py FreeSans.ttf 20 freesans_cyr_20.py -k cyrillic +font_to_py.py -x -k extended FreeSans.ttf 17 font10.py ``` -If a character set is specified, `--smallest` and `--largest` should not be -specified: these values are computed from the character set. - -The representation of non-contiguous character sets having large gaps (such as -the `extended` set) is not very efficient. This matters little if the font is -to be frozen as bytecode. I plan to investigate ways of improving this. +If a character set is specified via `-c` or `-k`, then `--smallest` and +`--largest` should not be specified: these values are computed from the +character set. Any requirement for arguments -xr will be specified in the device driver documentation. Bit reversal is required by some display hardware. -There have been reports that producing extended ASCII characters (ordinal -value > 127) from ttf files is unreliable. If the expected results are not -achieved, use an otf font. However I have successfully created the Cyrillic -font from a `ttf`. Perhaps not all fonts are created equal... +There have been reports that producing fonts with Unicode characters outside +the ASCII set from ttf files is unreliable. If expected results are not +achieved, use an otf font. I have successfully created Cyrillic and extended +fonts from a `ttf`, so I suspect the issue may be source fonts lacking the +required glyphs. The `-i` or `--iterate` argument. For specialist applications. Specifying this causes a generator function `glyphs` to be included in the Python font file. A @@ -153,7 +162,7 @@ My solution draws on the excellent example code written by Daniel Bader. This may be viewed [here](https://dbader.org/blog/monochrome-font-rendering-with-freetype-and-python) and [here](https://gist.github.com/dbader/5488053). -# Appendix: RAM utilisation Test Results +# Appendix 1: RAM utilisation Test Results The supplied `freesans20.py` and `courier20.py` files were frozen as bytecode on a Pyboard V1.0. The following code was pasted at the REPL: @@ -197,3 +206,16 @@ reclaimed on exit from the function. Its additional RAM use was 16 bytes. With a font of height 20 pixels RAM saving was an order of magnitude. The saving will be greater if larger fonts are used as RAM usage is independent of the array sizes. + +# Appendix 2: room for improvement + +The representation of non-contiguous character sets having large gaps (such as +the `extended` set) is not very efficient. This is because the index table +becomes sparse. This matters little if the font is to be frozen as bytecode +because the index is located in Flash rather than RAM. + +I have implemented a change which removes redundancy in the index file. Further +improvements would require a further level of indirection which would have the +drawback of increasing the size of small contiguous character sets - or +emitting two file formats with the same API. The latter does not appeal from a +support perspective. diff --git a/font_test.py b/font_test.py index 645fcf8..bf357dc 100644 --- a/font_test.py +++ b/font_test.py @@ -36,14 +36,14 @@ from font_to_py import Font, write_font def validate_hmap(data, height, width): bpr = (width - 1)//8 + 1 - msg = 'Horizontal map, invalid data length' - assert len(data) == bpr * height, msg + msg = 'Horizontal map, invalid data length got {} expected {}' + assert len(data) == bpr * height, msg.format(len(data), bpr * height) def validate_vmap(data, height, width): bpc = (height - 1)//8 + 1 - msg = 'Vertical map, invalid data length' - assert len(data) == bpc * width, msg + msg = 'Vertical map, invalid data length got {} expected {}' + assert len(data) == bpc * width, msg.format(len(data), bpc * width) # Routines to render to REPL @@ -135,10 +135,17 @@ def test_arrays(string, height, monospaced, hmap, reverse): # Render a string to REPL using a specified Python font file # usage font_test.test_font('freeserif', 'abc') -def test_font(fontfile, string): +# Default tests outliers with fonts created with -k extended +def test_font(fontfile, string='abg'+chr(126)+chr(127)+chr(176)+chr(177)+chr(937)+chr(981)): if fontfile in sys.modules: del sys.modules[fontfile] # force reload myfont = import_module(fontfile) + print(('Horizontal' if myfont.hmap() else 'Vertical') + ' map') + print(('Reverse' if myfont.reverse() else 'Normal') + ' bit order') + print(('Fixed' if myfont.monospaced() else 'Proportional') + ' spacing') + print('Dimensions height*max_width {} * {}'.format(myfont.height(), myfont.max_width())) + s, e = myfont.min_ch(), myfont.max_ch() + print('Start char "{}" (ord {}) end char "{}" (ord {})'.format(chr(s), s, chr(e), e)) height = myfont.height() for row in range(height): diff --git a/font_to_py.py b/font_to_py.py index c138dfd..c96f282 100755 --- a/font_to_py.py +++ b/font_to_py.py @@ -347,15 +347,13 @@ class Font(dict): index = bytearray() #((0, 0)) for char in self.charset: if char == '': - index += bytearray((0, 0, 0, 0)) - index[-1] = index[3] - index[-2] = index[2] + index += bytearray((0, 0)) else: index += (len(data)).to_bytes(2, byteorder='little') # Start width = self[char][1] data += (width).to_bytes(2, byteorder='little') data += bytearray(self.stream_char(char, hmap, reverse)) - index += (len(data)).to_bytes(2, byteorder='little') # End + index += (len(data)).to_bytes(2, byteorder='little') # End return data, index def build_binary_array(self, hmap, reverse, sig): @@ -367,23 +365,37 @@ class Font(dict): return data # PYTHON FILE WRITING +# Owing to sparse charsets and an index which only holds the atart of data, +# can't read next_offset but must calculate it STR01 = """# Code generated by font-to-py.py. # Font: {}{} # Cmd: {} -version = '0.27' +version = '0.28' + """ STR02 = """_mvfont = memoryview(_font) def get_ch(ch): ordch = ord(ch) - ordch = ordch + 1 if ordch >= {} and ordch <= {} else {} - idx_offs = 4 * (ordch - {}) + if ordch >= {0} and ordch <= {1}: + idx_offs = 2 * (ordch - {0} + 1) + else: + idx_offs = 0 offset = int.from_bytes(_index[idx_offs : idx_offs + 2], 'little') - next_offs = int.from_bytes(_index[idx_offs + 2 : idx_offs + 4], 'little') width = int.from_bytes(_font[offset:offset + 2], 'little') - return _mvfont[offset + 2:next_offs], {}, width +""" + +STR02H =""" + next_offs = offset + 2 + ((width - 1)//8 + 1) * {0} + return _mvfont[offset + 2:next_offs], {0}, width + +""" + +STR02V =""" + next_offs = offset + 2 + (({0} - 1)//8 + 1) * width + return _mvfont[offset + 2:next_offs], {0}, width """ @@ -422,7 +434,6 @@ def write_data(stream, fnt, font_path, hmap, reverse, iterate): st = '' if charset == '' else ' Char set: {}'.format(charset) cl = ' '.join(sys.argv) stream.write(STR01.format(os.path.split(font_path)[1], st, cl)) - stream.write('\n') write_func(stream, 'height', height) write_func(stream, 'max_width', fnt.max_width) write_func(stream, 'hmap', hmap) @@ -439,7 +450,11 @@ def write_data(stream, fnt, font_path, hmap, reverse, iterate): bw_index = ByteWriter(stream, '_index') bw_index.odata(index) bw_index.eot() - stream.write(STR02.format(minchar, maxchar, defchar, minchar, height)) + stream.write(STR02.format(minchar, maxchar, minchar)) + if hmap: + stream.write(STR02H.format(height)) + else: + stream.write(STR02V.format(height)) # BINARY OUTPUT # hmap reverse magic bytes diff --git a/writer/DRIVERS.md b/writer/DRIVERS.md index 72c5672..7cc3cc2 100644 --- a/writer/DRIVERS.md +++ b/writer/DRIVERS.md @@ -63,6 +63,7 @@ tested with `writer.py` and `nanogui.py`. * The [Nokia 5110](https://github.com/mcauser/micropython-pcd8544/blob/master/pcd8544_fb.py) * The [SSD1331 colour OLED](https://github.com/peterhinch/micropython-nano-gui/blob/master/drivers/ssd1331/ssd1331.py) * The [HX1230 96x68 LCD](https://github.com/mcauser/micropython-hx1230/blob/master/hx1230_fb.py) + * The [RA8875 driver for larger TFT displays](https://github.com/peterhinch/micropython_ra8875.git) The latter example illustrates a very simple driver which provides full access to `writer.py` and `nanogui.py` libraries. @@ -100,17 +101,18 @@ has the following outline definition (in practice the bytes objects are large): ```python # Code generated by font-to-py.py. -# Font: Arial.ttf -version = '0.25' +# Font: FreeSans.ttf +# Cmd: ./font_to_py.py -x FreeSans.ttf 17 font10.py +version = '0.28' def height(): - return 20 + return 17 def max_width(): - return 20 + return 17 def hmap(): - return False + return True def reverse(): return False @@ -125,20 +127,27 @@ def max_ch(): return 126 _font =\ -b'\x0b\x00\x18\x00\x00\x1c\x00\x00\x0e\x00\x00\x06\xce\x00\x06\xcf'\ -b'\x00\x86\x03\x00\xce\x01\x00\xfc\x00\x00\x38\x00\x00\x00\x00\x00'\ +b'\x09\x00\x3c\x00\xc7\x00\xc3\x00\x03\x00\x03\x00\x06\x00\x0c\x00'\ +b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'\ _index =\ -b'\x00\x00\x23\x00\x23\x00\x37\x00\x37\x00\x4b\x00\x4b\x00\x62\x00'\ -b'\x62\x00\x85\x00\x85\x00\xa8\x00\xa8\x00\xe0\x00\xe0\x00\x09\x01'\ +b'\x00\x00\x24\x00\x37\x00\x4a\x00\x5d\x00\x81\x00\xa5\x00\xc9\x00'\ +b'\xed\x00\x00\x01\x13\x01\x26\x01\x39\x01\x5d\x01\x70\x01\x83\x01'\ +b'\x60\x0b' _mvfont = memoryview(_font) def get_ch(ch): - # validate ch, if out of range use '?' - # get offsets into _font and retrieve char width - # Return: memoryview of bitmap, height and width - return mvfont[offset + 2, next_offset], height, width + ordch = ord(ch) + if ordch >= 32 and ordch <= 126: + idx_offs = 2 * (ordch - 32 + 1) + else: + idx_offs = 0 + offset = int.from_bytes(_index[idx_offs : idx_offs + 2], 'little') + width = int.from_bytes(_font[offset:offset + 2], 'little') + + next_offs = offset + 2 + ((width - 1)//8 + 1) * 17 + return _mvfont[offset + 2:next_offs], 17, width ``` `height` and `width` are specified in bits (pixels). See Appendix 1 for extra