9fec6ca398
* Fix CLOCK_REALTIME * Get Makefile working with both C and C++ compilers We need HAVE_STPCPY defined to avoid duplicating it. We shouldn't depend on CPPFLAGS if building with gcc. It may be necessary to build with a C++ compiler when using portaudio or integrating into a C++ project. Now it builds either way. * Build static lib by default; FT8_DEBUG for ASAN and debug symbols kiss_fft is omitted for now: not every program needs it, and the object file can still be linked separately, as before. `FT8_DEBUG=1 make` builds with `-fsanitize=address -ggdb3` and without `-O3` for debugging purposes. * copy_token(): save space for the null terminator If the given string is longer than the given length (available space), copy one less than the length and then append a null terminator. This guarantees that token will always be null-terminated, to avoid buffer overflow. For example in ftx_message_encode(), if message_text is excessively-long free text with no spaces, we attempt to copy it to call_to, which is 12 bytes long. We must copy only 11 bytes, so that it can be null-terminated. (In this case call_to is not really used anyway: it's not a callsign, and after failing the other encodings, we will call ftx_message_encode_free(msg, message_text), and then fail because it's too long. But don't crash on the way, either.) Discovered with ASAN. * Fix ftx_message_print() * Add support of non-standard callsigns with prefixes Closes kgoba/ft8_lib#44 * ftx_message_encode_free(): set all payload bytes; debug parsed words If a ftx_message_t struct is reused for all outgoing messages, it may have leftover i3.n3 values. Make sure to overwrite those. (Although, "telemetry" should really be type 0.5; but we are reusing ftx_message_encode_telemetry() in ftx_message_encode_free(), so I guess the caller who calls ftx_message_encode_telemetry() should set that byte itself.) Also output the "parsed '...' ..." debug log message before checking for errors, to help clarify why the error occurred. * Decode message field types and offsets The user of this library should not need to re-tokenize the plain-text message and guess again which field is a callsign, special-token, grid etc.: it's redundant and relatively failure-prone, whereas libft8 already has all the information. It already needed to decode individual tokens out of the bitstream and construct plain text from that. But some users (such as sbitx) color-code the message, and IMO it's best not to invent some ad-hoc markup language. So the message is kept as plain text, and its "metadata" is kept separate, in the form of a start position and a type enum for each recognizable span found in the message text. It could be argued that this could go further, such as splitting "CQ POTA" into two fields, FTX_FIELD_TOKEN and FTX_FIELD_TOKEN_ARG; however, the internal model so far that a std or nonstd message can contain only 3 fields, so we treat the whole thing as a "callsign", then pack28() detects CQ and calls parse_cq_modifier(). And FTX_MAX_MESSAGE_FIELDS is 3 because of this. Likewise, we could separate FTX_FIELD_CALL_TO and FTX_FIELD_CALL_DE: but they have positional meaning anyway. This could be reexamined later. * Use type 4 message for nonstandard CQ; get tests passing We need trim_brackets() for the tests, and it may be useful in user programs too. When a CHECK(this == that) test fails, it's helpful to see the values that were not equal; so add and use the CHECK_EQ_VAL(this, that) macro. A failed test needs to end with an extra newline to separate the output from the next test. And FAIL! is subtly better than FAIL: for searching in the terminal (also, I have muscle memory for it, from Qt work). * Fall back to free text to encode more than 3 tokens "TNX BOB 73 GL" (an example from WSJT-X tests) contains 4 tokens (which are not callsigns), but ft8_lib would have previously tried to compose a standard type 1 message. If a program that uses ft8_lib has paid attention to the field types and offsets from previous messages, and has UI for the user to select (for example) a FTX_FIELD_CALL field and start a new QSO, then it has enough information to call a type-specific function. pack_basecall() can be used to disambiguate whether the callsign is standard or not: then the program can call ftx_message_encode_std(), or ftx_message_encode_nonstd() if it detects a non-standard callsign or if ftx_message_encode_std() fails. Likewise, it can call ftx_message_encode_free() if it already knows the message must be sent as free text, or as a fallback if the other encode functions fail. ftx_message_encode() does this fallback sequence already, but it is not given field types to indicate what the message contains, so it could potentially make a mistake (e.g. with "THX BOB 73"). But it is retained for backwards compatibility, and also because it's easier to use; and now it gets more cases right than it did before. * Add pack_basecall() to public API A program could use it to pre-check whether a callsign is standard or not. (But if it's not standard, it may not be a callsign at all.) * Fix CQ_nnn and CQ_a[bcd] encoding Figured out how by reading subroutine pack28(c13,n28) in WSJT-X. The decoding already works, it seems. We now use space as the delimiter between CQ and the modifier, rather than underscore. An end user composing a message by hand would use space, so it must be detectable that way; and BTW CQ may also be a callsign prefix (Portugal, Azores, Madeira), so we cannot simply detect CQ without checking the following delimiter, and assume that it's a CQ message. nnn and a[bcd] are unlike callsigns (a CQ modifier can have letters OR numbers, but not both, whereas a callsign ALWAYS has both); so we can reliably detect modifiers without needing underscore as a delimiter. --------- Co-authored-by: Georgy Dyuldin <g.dyuldin@gmail.com> |
||
|---|---|---|
| common | ||
| demo | ||
| fft | ||
| ft4_ft8_public | ||
| ft8 | ||
| test | ||
| utils | ||
| .clang-format | ||
| .gitignore | ||
| LICENSE | ||
| Makefile | ||
| README.md | ||
README.md
FT8 (and now FT4) library
C implementation of a lightweight FT8/FT4 decoder and encoder, mostly intended for experimental use on microcontrollers.
The intent of this library is to allow FT8/FT4 encoding and decoding in standalone environments (i.e. without a PC or RPi), e.g. automated beacons or SDR transceivers. It's also my learning process, optimization problem and source of fun.
The encoding process is relatively light on resources, and an Arduino should be perfectly capable of running this code.
The decoder is designed with memory and computing efficiency in mind, in order to be usable with a fast enough microcontroller. It is shown to be working on STM32F7 boards fast enough for real work, but the embedded application itself is beyond this repository. This repository provides an example decoder which can decode a 15-second WAV file on a desktop machine or SBC. The decoder needs to access the whole 15-second window in spectral magnitude representation (the window can be also shorter, and messages can have varying starting time within the window). The example FT8 decoder can work with slightly less than 200 KB of RAM.
Current state
Currently the basic message set for establishing QSOs, as well as telemetry and free-text message modes are supported:
- CQ {call} {grid}, e.g. CQ CA0LL GG77
- CQ {xy} {call} {grid}, e.g. CQ JA CA0LL GG77
- {call} {call} {report}, e.g. CA0LL OT7ER R-07
- {call} {call} 73/RRR/RR73, e.g. OT7ER CA0LL 73
- Free-text messages (up to 13 characters from a limited alphabet) (decoding only, untested)
- Telemetry data (71 bits as 18 hex symbols)
Encoding and decoding works for both FT8 and FT4. For encoding and decoding, there is a console application provided for each, which serves mostly as test code, and could be a starting point for your potential application on an MCU. The console apps should run perfectly well on a RPi or a PC/Mac. I don't provide a concrete example for a particular MCU hardware here, since it would be very specific.
The code is not yet really a library, rather a collection of routines and example code.
Future ideas
Incremental decoding (processing during the 15 second window) is something that I would like to explore, but haven't started.
These features are low on my priority list:
- Contest modes
- Compound callsigns with country prefixes and special callsigns
What to do with it
You can generate 15-second WAV files with your own messages as a proof of concept or for testing purposes. They can either be played back or opened directly from WSJT-X. To do that, run make. Then run gen_ft8 (run it without parameters to check what parameters are supported). Currently messages are modulated at 1000-1050 Hz.
You can decode 15-second (or shorter) WAV files with decode_ft8. This is only an example application and does not support live processing/recording. For that you could use third party code (PortAudio, for example).
References and credits
Thanks goes out to:
- my contributors who have provided me with various improvements which have often been beyond my skill set.
- Robert Morris, AB1HL, whose Python code (https://github.com/rtmrtmrtmrtm/weakmon) inspired this and helped to test various parts of the code.
- Mark Borgerding for his FFT implementation (https://github.com/mborgerding/kissfft). I have included a portion of his code.
- WSJT-X authors, who developed a very interesting and novel communications protocol
The details of FT4 and FT8 procotols and decoding/encoding are described here: https://physics.princeton.edu/pulsar/k1jt/FT4_FT8_QEX.pdf
The public part of FT4/FT8 implementation is included in this repository under ft4_ft8_public.
Of course in moments of frustration I have looked up the original WSJT-X code, which is mostly written in Fortran (http://physics.princeton.edu/pulsar/K1JT/wsjtx.html). However, this library contains my own original DSP routines and a different implementation of the decoder which is suitable for resource-constrained embedded environments.
Karlis Goba, YL3JG