V0.28 release.

FONT_TO_PY.md

@@ -5,12 +5,21 @@ is to save RAM on resource-limited targets: the font file may be incorporated
 into a firmware build such that it occupies flash memory rather than scarce
 RAM. Python code built into firmware is known as frozen bytecode.
 
+## V0.27/0.28 notes
+
+7 Sept 2019
+
+Remove redundancy from index file: significantly reduces file size for sparse
+fonts. Add a comment field in the output file showing creation command line.
+Repo includes the file `extended`. This facilitates creating fonts with useful
+scientific glyphs. Improvements to `font_test.py`.
+
 ###### [Main README](./README.md)
 
 # Dependencies
 
 The utility requires Python 3.2 or greater, also `freetype` which may be
-installed using `pip3`. On Linux at a root prompt:
+installed using `pip3`. On Linux (you may need a root prompt):
 
 ```shell
 # apt-get install python3-pip
@@ -25,10 +34,11 @@ required height in pixels and outputs a Python 3 source file. The pixel layout
 is determined by command arguments. By default fonts are stored in variable
 pitch form. This may be overidden by a command line argument.
 
-By default the ASCII character set (ordinal values 32 to 126 inclusive) is
+By default the printable ASCII character set (ordinal values 32 to 126
-supported. Command line arguments can modify this range as required, if
+inclusive) is supported (i.e. not including control characters). Command line
-necessary to include extended ASCII characters up to 255. Alternatively non
+arguments can modify this range as required to specify arbitrary sets of
-English or non-contiguous character sets may be defined.
+Unicode characters. Non-English and non-contiguous character sets may be
+defined.
 
 Further arguments ensure that the byte contents and layout are correct for the
 target display hardware. Their usage should be specified in the documentation
@@ -55,7 +65,7 @@ Example usage to produce a file `myfont.py` with height of 23 pixels:
  32 (ASCII space).
  * -l or --largest Ordinal value of largest character to be stored. Default 126.
  * -e or --errchar Ordinal value of character to be rendered if an attempt is
- made to display an out-of-range character. Default 63 (ASCII "?").
+ made to display an out-of-range character. Default 63 (ord("?")).
  * -i or --iterate Specialist use. See below.
  * -c or --charset Option to restrict the characters in the font to a specific
  set. See below.
@@ -63,8 +73,8 @@ Example usage to produce a file `myfont.py` with height of 23 pixels:
  for alternative character sets such as Cyrillic: the file must contain the
  character set to be included. An example file is `cyrillic`. Another is 
  `extended` which adds unicode characters "° μ π ω ϕ θ α β γ δ λ Ω" to those
- with `ord` values from 32-128. Such files will only produce useful results if
+ with `ord` values from 32-126. Such files will only produce useful results if
- the font file includes them.
+ the source font file includes those glyphs.
 
 The -c option may be used to reduce the size of the font file by limiting the
 character set. If the font file is frozen as bytecode this will not reduce RAM
@@ -76,22 +86,21 @@ $ font_to_py.py Arial.ttf 20 arial_clock.py -c 1234567890:
 Example usage with the -k option:  
 ```shell
 font_to_py.py FreeSans.ttf 20 freesans_cyr_20.py -k cyrillic
+font_to_py.py -x -k extended FreeSans.ttf 17 font10.py
 ```
 
-If a character set is specified, `--smallest` and `--largest` should not be
+If a character set is specified via `-c` or `-k`, then `--smallest` and
-specified: these values are computed from the character set.
+`--largest` should not be specified: these values are computed from the
-
+character set.
-The representation of non-contiguous character sets having large gaps (such as
-the `extended` set) is not very efficient. This matters little if the font is
-to be frozen as bytecode. I plan to investigate ways of improving this.
 
 Any requirement for arguments -xr will be specified in the device driver
 documentation. Bit reversal is required by some display hardware.
 
-There have been reports that producing extended ASCII characters (ordinal
+There have been reports that producing fonts with Unicode characters outside
-value > 127) from ttf files is unreliable. If the expected results are not
+the ASCII set from ttf files is unreliable. If expected results are not
-achieved, use an otf font. However I have successfully created the Cyrillic
+achieved, use an otf font. I have successfully created Cyrillic and extended
-font from a `ttf`. Perhaps not all fonts are created equal...
+fonts from a `ttf`, so I suspect the issue may be source fonts lacking the
+required glyphs.
 
 The `-i` or `--iterate` argument. For specialist applications. Specifying this
 causes a generator function `glyphs` to be included in the Python font file. A
@@ -153,7 +162,7 @@ My solution draws on the excellent example code written by Daniel Bader. This
 may be viewed [here](https://dbader.org/blog/monochrome-font-rendering-with-freetype-and-python)
 and [here](https://gist.github.com/dbader/5488053).
 
-# Appendix: RAM utilisation Test Results
+# Appendix 1: RAM utilisation Test Results
 
 The supplied `freesans20.py` and `courier20.py` files were frozen as bytecode
 on a Pyboard V1.0. The following code was pasted at the REPL:
@@ -197,3 +206,16 @@ reclaimed on exit from the function. Its additional RAM use was 16 bytes.
 With a font of height 20 pixels RAM saving was an order of magnitude. The
 saving will be greater if larger fonts are used as RAM usage is independent of
 the array sizes.
+
+# Appendix 2: room for improvement
+
+The representation of non-contiguous character sets having large gaps (such as
+the `extended` set) is not very efficient. This is because the index table
+becomes sparse. This matters little if the font is to be frozen as bytecode
+because the index is located in Flash rather than RAM.
+
+I have implemented a change which removes redundancy in the index file. Further
+improvements would require a further level of indirection which would have the
+drawback of increasing the size of small contiguous character sets - or
+emitting two file formats with the same API. The latter does not appeal from a
+support perspective.

font_test.py

@@ -36,14 +36,14 @@ from font_to_py import Font, write_font
 
 def validate_hmap(data, height, width):
     bpr = (width - 1)//8 + 1
-    msg = 'Horizontal map, invalid data length'
+    msg = 'Horizontal map, invalid data length got {} expected {}'
-    assert len(data) == bpr * height, msg
+    assert len(data) == bpr * height, msg.format(len(data), bpr * height)
 
 
 def validate_vmap(data, height, width):
     bpc = (height - 1)//8 + 1
-    msg = 'Vertical map, invalid data length'
+    msg = 'Vertical map, invalid data length got {} expected {}'
-    assert len(data) == bpc * width, msg
+    assert len(data) == bpc * width, msg.format(len(data), bpc * width)
 
 
 # Routines to render to REPL
@@ -135,10 +135,17 @@ def test_arrays(string, height, monospaced, hmap, reverse):
 
 # Render a string to REPL using a specified Python font file
 # usage font_test.test_font('freeserif', 'abc')
-def test_font(fontfile, string):
+# Default tests outliers with fonts created with -k extended
+def test_font(fontfile, string='abg'+chr(126)+chr(127)+chr(176)+chr(177)+chr(937)+chr(981)):
     if fontfile in sys.modules:
         del sys.modules[fontfile]  # force reload
     myfont = import_module(fontfile)
+    print(('Horizontal' if myfont.hmap() else 'Vertical') + ' map')
+    print(('Reverse' if myfont.reverse() else 'Normal') + ' bit order')
+    print(('Fixed' if myfont.monospaced() else 'Proportional') + ' spacing')
+    print('Dimensions height*max_width {} * {}'.format(myfont.height(), myfont.max_width()))
+    s, e = myfont.min_ch(), myfont.max_ch()
+    print('Start char "{}" (ord {}) end char "{}" (ord {})'.format(chr(s), s, chr(e), e))
 
     height = myfont.height()
     for row in range(height):

font_to_py.py

@@ -347,9 +347,7 @@ class Font(dict):
         index = bytearray() #((0, 0))
         for char in self.charset:
             if char == '':
-                index += bytearray((0, 0, 0, 0))
+                index += bytearray((0, 0))
-                index[-1] = index[3]
-                index[-2] = index[2]
             else:
                 index += (len(data)).to_bytes(2, byteorder='little')  # Start
                 width = self[char][1]
@@ -367,23 +365,37 @@ class Font(dict):
         return data
 
 # PYTHON FILE WRITING
+# Owing to sparse charsets and an index which only holds the atart of data,
+# can't read next_offset but must calculate it
 
 STR01 = """# Code generated by font-to-py.py.
 # Font: {}{}
 # Cmd: {}
-version = '0.27'
+version = '0.28'
+
 """
 
 STR02 = """_mvfont = memoryview(_font)
 
 def get_ch(ch):
     ordch = ord(ch)
-    ordch = ordch + 1 if ordch >= {} and ordch <= {} else {}
+    if ordch >= {0} and ordch <= {1}:
-    idx_offs = 4 * (ordch - {})
+        idx_offs = 2 * (ordch - {0} + 1)
+    else:
+        idx_offs = 0
     offset = int.from_bytes(_index[idx_offs : idx_offs + 2], 'little')
-    next_offs = int.from_bytes(_index[idx_offs + 2 : idx_offs + 4], 'little')
     width = int.from_bytes(_font[offset:offset + 2], 'little')
-    return _mvfont[offset + 2:next_offs], {}, width
+"""
+
+STR02H ="""
+    next_offs = offset + 2 + ((width - 1)//8 + 1) * {0}
+    return _mvfont[offset + 2:next_offs], {0}, width
+ 
+"""
+
+STR02V ="""
+    next_offs = offset + 2 + (({0} - 1)//8 + 1) * width
+    return _mvfont[offset + 2:next_offs], {0}, width
  
 """
 
@@ -422,7 +434,6 @@ def write_data(stream, fnt, font_path, hmap, reverse, iterate):
     st = '' if charset == '' else ' Char set: {}'.format(charset)
     cl = ' '.join(sys.argv)
     stream.write(STR01.format(os.path.split(font_path)[1], st, cl))
-    stream.write('\n')
     write_func(stream, 'height', height)
     write_func(stream, 'max_width', fnt.max_width)
     write_func(stream, 'hmap', hmap)
@@ -439,7 +450,11 @@ def write_data(stream, fnt, font_path, hmap, reverse, iterate):
     bw_index = ByteWriter(stream, '_index')
     bw_index.odata(index)
     bw_index.eot()
-    stream.write(STR02.format(minchar, maxchar, defchar, minchar, height))
+    stream.write(STR02.format(minchar, maxchar, minchar))
+    if hmap:
+        stream.write(STR02H.format(height))
+    else:
+        stream.write(STR02V.format(height))
 
 # BINARY OUTPUT
 # hmap reverse magic bytes

writer/DRIVERS.md

@@ -63,6 +63,7 @@ tested with `writer.py` and `nanogui.py`.
  * The [Nokia 5110](https://github.com/mcauser/micropython-pcd8544/blob/master/pcd8544_fb.py)
  * The [SSD1331 colour OLED](https://github.com/peterhinch/micropython-nano-gui/blob/master/drivers/ssd1331/ssd1331.py)
  * The [HX1230 96x68 LCD](https://github.com/mcauser/micropython-hx1230/blob/master/hx1230_fb.py)
+ * The [RA8875 driver for larger TFT displays](https://github.com/peterhinch/micropython_ra8875.git)
 
 The latter example illustrates a very simple driver which provides full access
 to `writer.py` and `nanogui.py` libraries.
@@ -100,17 +101,18 @@ has the following outline definition (in practice the bytes objects are large):
 
 ```python
 # Code generated by font-to-py.py.
-# Font: Arial.ttf
+# Font: FreeSans.ttf
-version = '0.25'
+# Cmd: ./font_to_py.py -x FreeSans.ttf 17 font10.py
+version = '0.28'
 
 def height():
-    return 20
+    return 17
 
 def max_width():
-    return 20
+    return 17
 
 def hmap():
-    return False
+    return True
 
 def reverse():
     return False
@@ -125,20 +127,27 @@ def max_ch():
     return 126
 
 _font =\
-b'\x0b\x00\x18\x00\x00\x1c\x00\x00\x0e\x00\x00\x06\xce\x00\x06\xcf'\
+b'\x09\x00\x3c\x00\xc7\x00\xc3\x00\x03\x00\x03\x00\x06\x00\x0c\x00'\
-b'\x00\x86\x03\x00\xce\x01\x00\xfc\x00\x00\x38\x00\x00\x00\x00\x00'\
+b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'\
 
 _index =\
-b'\x00\x00\x23\x00\x23\x00\x37\x00\x37\x00\x4b\x00\x4b\x00\x62\x00'\
+b'\x00\x00\x24\x00\x37\x00\x4a\x00\x5d\x00\x81\x00\xa5\x00\xc9\x00'\
-b'\x62\x00\x85\x00\x85\x00\xa8\x00\xa8\x00\xe0\x00\xe0\x00\x09\x01'\
+b'\xed\x00\x00\x01\x13\x01\x26\x01\x39\x01\x5d\x01\x70\x01\x83\x01'\
+b'\x60\x0b'
 
 _mvfont = memoryview(_font)
 
 def get_ch(ch):
-    # validate ch, if out of range use '?'
+    ordch = ord(ch)
-    # get offsets into _font and retrieve char width
+    if ordch >= 32 and ordch <= 126:
-    # Return: memoryview of bitmap, height and width
+        idx_offs = 2 * (ordch - 32 + 1)
-    return mvfont[offset + 2, next_offset], height, width
+    else:
+        idx_offs = 0
+    offset = int.from_bytes(_index[idx_offs : idx_offs + 2], 'little')
+    width = int.from_bytes(_font[offset:offset + 2], 'little')
+
+    next_offs = offset + 2 + ((width - 1)//8 + 1) * 17
+    return _mvfont[offset + 2:next_offs], 17, width
 ```
 
 `height` and `width` are specified in bits (pixels). See Appendix 1 for extra