Merge pull request #25 from etherkit/v1.3.0

V1.3.0
master v1.3.0
Jason Milldrum 2021-06-30 18:00:15 -07:00 zatwierdzone przez GitHub
commit a787ff00f3
Nie znaleziono w bazie danych klucza dla tego podpisu
ID klucza GPG: 4AEE18F83AFDEB23
7 zmienionych plików z 710 dodań i 70 usunięć

Wyświetl plik

@ -0,0 +1,12 @@
name: arduino-lint
on: [push, pull_request]
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: arduino/arduino-lint-action@v1.0.0
with:
library-manager: update
compliance: specification

Wyświetl plik

@ -1,7 +1,7 @@
JT65/JT9/JT4/FT8/WSPR/FSQ Encoder Library for Arduino
=====================================================
This library very simply generates a set of channel symbols for JT65, JT9, JT4, FT8, or WSPR based on the user providing a properly formatted Type 6 message for JT65, JT9, or JT4 (which is 13 valid characters), Type 0.0 or 0.5 message for FT8 (v2.0.0 protocol) or a callsign, Maidenhead grid locator, and power output for WSPR. It will also generate an arbitrary FSQ message of up to 200 characters in both directed and non-directed format. When paired with a synthesizer that can output frequencies in fine, phase-continuous tuning steps (such as the Si5351), then a beacon or telemetry transmitter can be created which can change the transmitted characters as needed from the Arduino.
This library very simply generates a set of channel symbols for JT65, JT9, JT4, FT8, or WSPR based on the user providing a properly formatted Type 6 message for JT65, JT9, or JT4 (which is 13 valid characters), Type 0.0 or 0.5 message for FT8 (v2.0.0 protocol) or a Type 1, Type 2, or Type 3 message for WSPR. It will also generate an arbitrary FSQ message of up to 200 characters in both directed and non-directed format. When paired with a synthesizer that can output frequencies in fine, phase-continuous tuning steps (such as the Si5351), then a beacon or telemetry transmitter can be created which can change the transmitted characters as needed from the Arduino.
Please feel free to use the issues feature of GitHub if you run into problems or have suggestions for important features to implement.
@ -23,6 +23,20 @@ RAM Usage
---------
Most of the encoding functions need to manipulate multiple arrays of symbols in RAM at the same time, and therefore are quite RAM intensive. Care has been taken to put as much data into program memory as is possible, but the encoding functions still can cause problems with the low RAM microcontrollers such as the ATmegaxx8 series. If you are using these, then please be sure to call them only once when a transmit buffer needs to be created or changed, and call them separately of other subroutine calls. When using other microcontrollers that have more RAM, such as most of the ARM ICs, this won't be as much of a problem. If you see unusual freezes, that almost certainly indicates a RAM shortage.
WSPR Messages
-------------
JTEncode includes support for all three WSPR message types. A brief listing of the three types is given below:
| Message Type | Fields | Example |
|--------------|--------|---------|
| Type 1 | Callsign, Grid (4 digit), Power | NT7S CN85 30 |
| Type 2 | Callsign with prefix or suffix, Power | NT7S/P 30 |
| Type 3 | Callsign Hash, Grid (6 digit), Power | \<NT7S\> CN85NM 30 |
Most WSPR messages are type 1, however sometimes type 2 and 3 messages are needed. Type 2 messages allow you to send a callsign with a prefix of up to three characters, a suffix of a single character, or a suffix consisting of two numerical digits. Type 3 messages are typically used in conjunction with type 2 messages since type 2 messages don't include a grid locator. The type 3 message sends a 15-bit hash of the included callsign, along with a 6 digit grid locator and the power.
Type 2 messages can be sent in JTEncode simply by including the slashed prefix or suffix in the callsign field. A type 3 message can be sent by enclosing a callsign with angle brackets (as seen in the example above).
Example
-------
There is a simple example that is placed in your examples menu under JTEncode. Open this to see how to incorporate this library with your code. The example provided with with the library is meant to be used in conjunction with the [Etherkit Si5351A Breakout Board](https://www.etherkit.com/rf-modules/si5351a-breakout-board.html), although it could be modified to use with other synthesizers which meet the technical requirements of the JT65/JT9/JT4/WSPR/FSQ modes.
@ -185,10 +199,11 @@ Public Methods
/*
* wspr_encode(const char * call, const char * loc, const uint8_t dbm, uint8_t * symbols)
*
* Takes an arbitrary message of up to 13 allowable characters and returns
* Takes a callsign, grid locator, and power level and returns a WSPR symbol
* table for a Type 1, 2, or 3 message.
*
* call - Callsign (6 characters maximum).
* loc - Maidenhead grid locator (4 characters maximum).
* call - Callsign (12 characters maximum).
* loc - Maidenhead grid locator (6 characters maximum).
* dbm - Output power in dBm.
* symbols - Array of channel symbols to transmit returned by the method.
* Ensure that you pass a uint8_t array of at least size WSPR_SYMBOL_COUNT to the method.
@ -262,6 +277,10 @@ Also, a big thank you to Murray Greenman, ZL1BPU for working allowing me to pick
Changelog
---------
* v1.3.0
* WSPR Type 2 and Type 3 message capability added
* v1.2.1
* Fix keywords.txt
@ -297,6 +316,10 @@ Changelog
* Initial Release
Arduino Lint Status
-------------------
[![arduino-lint Actions Status](https://github.com/etherkit/JTEncode/workflows/arduino-lint/badge.svg)](https://github.com/etherkit/JTEncode/actions)
License
-------

Wyświetl plik

@ -1,5 +1,5 @@
name=Etherkit JTEncode
version=1.2.1
version=1.3.0
author=Jason Milldrum <milldrum@gmail.com>
maintainer=Jason Milldrum <milldrum@gmail.com>
sentence=Generate JT65, JT9, JT4, FT8, WSPR, and FSQ symbols on your Arduino.

Wyświetl plik

@ -1,7 +1,7 @@
/*
* JTEncode.cpp - JT65/JT9/WSPR/FSQ encoder library for Arduino
*
* Copyright (C) 2015-2018 Jason Milldrum <milldrum@gmail.com>
* Copyright (C) 2015-2021 Jason Milldrum <milldrum@gmail.com>
*
* Based on the algorithms presented in the WSJT software suite.
* Thanks to Andy Talbot G4JNT for the whitepaper on the WSPR encoding
@ -24,6 +24,7 @@
#include <JTEncode.h>
#include <crc14.h>
#include <generator.h>
#include <nhash.h>
#include <string.h>
#include <ctype.h>
@ -48,6 +49,7 @@ JTEncode::JTEncode(void)
{
// Initialize the Reed-Solomon encoder
rs_inst = (struct rs *)(intptr_t)init_rs_int(6, 0x43, 3, 1, 51, 0);
// memset(callsign, 0, 13);
}
/*
@ -188,19 +190,20 @@ void JTEncode::jt4_encode(const char * msg, uint8_t * symbols)
/*
* wspr_encode(const char * call, const char * loc, const uint8_t dbm, uint8_t * symbols)
*
* Takes an arbitrary message of up to 13 allowable characters and returns
* Takes a callsign, grid locator, and power level and returns a WSPR symbol
* table for a Type 1, 2, or 3 message.
*
* call - Callsign (6 characters maximum).
* loc - Maidenhead grid locator (4 characters maximum).
* call - Callsign (12 characters maximum).
* loc - Maidenhead grid locator (6 characters maximum).
* dbm - Output power in dBm.
* symbols - Array of channel symbols to transmit returned by the method.
* Ensure that you pass a uint8_t array of at least size WSPR_SYMBOL_COUNT to the method.
*
*/
void JTEncode::wspr_encode(const char * call, const char * loc, const uint8_t dbm, uint8_t * symbols)
void JTEncode::wspr_encode(const char * call, const char * loc, const int8_t dbm, uint8_t * symbols)
{
char call_[7];
char loc_[5];
char call_[13];
char loc_[7];
uint8_t dbm_ = dbm;
strcpy(call_, call);
strcpy(loc_, loc);
@ -533,7 +536,7 @@ uint8_t JTEncode::ft_code(char c)
uint8_t JTEncode::wspr_code(char c)
{
// Validate the input then return the proper integer code.
// Return 255 as an error code if the char is not allowed.
// Change character to a space if the char is not allowed.
if(isdigit(c))
{
@ -549,7 +552,7 @@ uint8_t JTEncode::wspr_code(char c)
}
else
{
return 255;
return 36;
}
}
@ -575,12 +578,10 @@ void JTEncode::jt_message_prep(char * message)
// Pad the message with trailing spaces
uint8_t len = strlen(message);
if(len < 13)
for(i = len; i < 13; i++)
{
for(i = len; i <= 13; i++)
{
message[i] = ' ';
}
message[i] = ' ';
}
// Convert all chars to uppercase
@ -612,53 +613,65 @@ void JTEncode::ft_message_prep(char * message)
strcpy(message, temp_msg);
}
void JTEncode::wspr_message_prep(char * call, char * loc, uint8_t dbm)
void JTEncode::wspr_message_prep(char * call, char * loc, int8_t dbm)
{
// Callsign validation and padding
// -------------------------------
// If only the 2nd character is a digit, then pad with a space.
// If this happens, then the callsign will be truncated if it is
// longer than 5 characters.
if((call[1] >= '0' && call[1] <= '9') && (call[2] < '0' || call[2] > '9'))
{
memmove(call + 1, call, 5);
call[0] = ' ';
}
// Now the 3rd charcter in the callsign must be a digit
if(call[2] < '0' || call[2] > '9')
{
// TODO: need a better way to handle this
call[2] = '0';
}
// Ensure that the only allowed characters are digits and
// uppercase letters
// Ensure that the only allowed characters are digits, uppercase letters, slash, and angle brackets
uint8_t i;
for(i = 0; i < 6; i++)
for(i = 0; i < 12; i++)
{
call[i] = toupper(call[i]);
if(!(isdigit(call[i]) || isupper(call[i])))
if(call[i] != '/' && call[i] != '<' && call[i] != '>')
{
call[i] = ' ';
call[i] = toupper(call[i]);
if(!(isdigit(call[i]) || isupper(call[i])))
{
call[i] = ' ';
}
}
}
call[12] = 0;
memcpy(callsign, call, 6);
strncpy(callsign, call, 12);
// Grid locator validation
for(i = 0; i < 4; i++)
if(strlen(loc) == 4 || strlen(loc) == 6)
{
loc[i] = toupper(loc[i]);
if(!(isdigit(loc[i]) || (loc[i] >= 'A' && loc[i] <= 'R')))
for(i = 0; i <= 1; i++)
{
memcpy(loc, "AA00", 5);
//loc = "AA00";
loc[i] = toupper(loc[i]);
if((loc[i] < 'A' || loc[i] > 'R'))
{
strncpy(loc, "AA00AA", 7);
}
}
for(i = 2; i <= 3; i++)
{
if(!(isdigit(loc[i])))
{
strncpy(loc, "AA00AA", 7);
}
}
}
else
{
strncpy(loc, "AA00AA", 7);
}
if(strlen(loc) == 6)
{
for(i = 4; i <= 5; i++)
{
loc[i] = toupper(loc[i]);
if((loc[i] < 'A' || loc[i] > 'X'))
{
strncpy(loc, "AA00AA", 7);
}
}
}
memcpy(locator, loc, 4);
strncpy(locator, loc, 7);
// Power level validation
// Only certain increments are allowed
@ -666,10 +679,12 @@ void JTEncode::wspr_message_prep(char * call, char * loc, uint8_t dbm)
{
dbm = 60;
}
const uint8_t valid_dbm[19] =
{0, 3, 7, 10, 13, 17, 20, 23, 27, 30, 33, 37, 40,
const uint8_t VALID_DBM_SIZE = 28;
const int8_t valid_dbm[VALID_DBM_SIZE] =
{-30, -27, -23, -20, -17, -13, -10, -7, -3,
0, 3, 7, 10, 13, 17, 20, 23, 27, 30, 33, 37, 40,
43, 47, 50, 53, 57, 60};
for(i = 0; i < 19; i++)
for(i = 0; i < VALID_DBM_SIZE; i++)
{
if(dbm == valid_dbm[i])
{
@ -677,7 +692,7 @@ void JTEncode::wspr_message_prep(char * call, char * loc, uint8_t dbm)
}
}
// If we got this far, we have an invalid power level, so we'll round down
for(i = 1; i < 19; i++)
for(i = 1; i < VALID_DBM_SIZE; i++)
{
if(dbm < valid_dbm[i] && dbm >= valid_dbm[i - 1])
{
@ -794,18 +809,185 @@ void JTEncode::wspr_bit_packing(uint8_t * c)
{
uint32_t n, m;
n = wspr_code(callsign[0]);
n = n * 36 + wspr_code(callsign[1]);
n = n * 10 + wspr_code(callsign[2]);
n = n * 27 + (wspr_code(callsign[3]) - 10);
n = n * 27 + (wspr_code(callsign[4]) - 10);
n = n * 27 + (wspr_code(callsign[5]) - 10);
// Determine if type 1, 2 or 3 message
char* slash_avail = strchr(callsign, (int)'/');
if(callsign[0] == '<')
{
// Type 3 message
char base_call[13];
memset(base_call, 0, 13);
uint32_t init_val = 146;
char* bracket_avail = strchr(callsign, (int)'>');
int call_len = bracket_avail - callsign - 1;
strncpy(base_call, callsign + 1, call_len);
uint32_t hash = nhash_(base_call, &call_len, &init_val);
hash &= 32767;
m = ((179 - 10 * (locator[0] - 'A') - (locator[2] - '0')) * 180) +
(10 * (locator[1] - 'A')) + (locator[3] - '0');
m = (m * 128) + power + 64;
// Convert 6 char grid square to "callsign" format for transmission
// by putting the first character at the end
char temp_loc = locator[0];
locator[0] = locator[1];
locator[1] = locator[2];
locator[2] = locator[3];
locator[3] = locator[4];
locator[4] = locator[5];
locator[5] = temp_loc;
// Callsign is 28 bits, locator/power is 22 bits.
n = wspr_code(locator[0]);
n = n * 36 + wspr_code(locator[1]);
n = n * 10 + wspr_code(locator[2]);
n = n * 27 + (wspr_code(locator[3]) - 10);
n = n * 27 + (wspr_code(locator[4]) - 10);
n = n * 27 + (wspr_code(locator[5]) - 10);
m = (hash * 128) - (power + 1) + 64;
}
else if(slash_avail == (void *)0)
{
// Type 1 message
pad_callsign(callsign);
n = wspr_code(callsign[0]);
n = n * 36 + wspr_code(callsign[1]);
n = n * 10 + wspr_code(callsign[2]);
n = n * 27 + (wspr_code(callsign[3]) - 10);
n = n * 27 + (wspr_code(callsign[4]) - 10);
n = n * 27 + (wspr_code(callsign[5]) - 10);
m = ((179 - 10 * (locator[0] - 'A') - (locator[2] - '0')) * 180) +
(10 * (locator[1] - 'A')) + (locator[3] - '0');
m = (m * 128) + power + 64;
}
else if(slash_avail)
{
// Type 2 message
int slash_pos = slash_avail - callsign;
uint8_t i;
// Determine prefix or suffix
if(callsign[slash_pos + 2] == ' ' || callsign[slash_pos + 2] == 0)
{
// Single character suffix
char base_call[7];
memset(base_call, 0, 7);
strncpy(base_call, callsign, slash_pos);
for(i = 0; i < 7; i++)
{
base_call[i] = toupper(base_call[i]);
if(!(isdigit(base_call[i]) || isupper(base_call[i])))
{
base_call[i] = ' ';
}
}
pad_callsign(base_call);
n = wspr_code(base_call[0]);
n = n * 36 + wspr_code(base_call[1]);
n = n * 10 + wspr_code(base_call[2]);
n = n * 27 + (wspr_code(base_call[3]) - 10);
n = n * 27 + (wspr_code(base_call[4]) - 10);
n = n * 27 + (wspr_code(base_call[5]) - 10);
char x = callsign[slash_pos + 1];
if(x >= 48 && x <= 57)
{
x -= 48;
}
else if(x >= 65 && x <= 90)
{
x -= 55;
}
else
{
x = 38;
}
m = 60000 - 32768 + x;
m = (m * 128) + power + 2 + 64;
}
else if(callsign[slash_pos + 3] == ' ' || callsign[slash_pos + 3] == 0)
{
// Two-digit numerical suffix
char base_call[7];
memset(base_call, 0, 7);
strncpy(base_call, callsign, slash_pos);
for(i = 0; i < 6; i++)
{
base_call[i] = toupper(base_call[i]);
if(!(isdigit(base_call[i]) || isupper(base_call[i])))
{
base_call[i] = ' ';
}
}
pad_callsign(base_call);
n = wspr_code(base_call[0]);
n = n * 36 + wspr_code(base_call[1]);
n = n * 10 + wspr_code(base_call[2]);
n = n * 27 + (wspr_code(base_call[3]) - 10);
n = n * 27 + (wspr_code(base_call[4]) - 10);
n = n * 27 + (wspr_code(base_call[5]) - 10);
// TODO: needs validation of digit
m = 10 * (callsign[slash_pos + 1] - 48) + callsign[slash_pos + 2] - 48;
m = 60000 + 26 + m;
m = (m * 128) + power + 2 + 64;
}
else
{
// Prefix
char prefix[4];
char base_call[7];
memset(prefix, 0, 4);
memset(base_call, 0, 7);
strncpy(prefix, callsign, slash_pos);
strncpy(base_call, callsign + slash_pos + 1, 7);
if(prefix[2] == ' ' || prefix[2] == 0)
{
// Right align prefix
prefix[3] = 0;
prefix[2] = prefix[1];
prefix[1] = prefix[0];
prefix[0] = ' ';
}
for(uint8_t i = 0; i < 6; i++)
{
base_call[i] = toupper(base_call[i]);
if(!(isdigit(base_call[i]) || isupper(base_call[i])))
{
base_call[i] = ' ';
}
}
pad_callsign(base_call);
n = wspr_code(base_call[0]);
n = n * 36 + wspr_code(base_call[1]);
n = n * 10 + wspr_code(base_call[2]);
n = n * 27 + (wspr_code(base_call[3]) - 10);
n = n * 27 + (wspr_code(base_call[4]) - 10);
n = n * 27 + (wspr_code(base_call[5]) - 10);
m = 0;
for(uint8_t i = 0; i < 3; ++i)
{
m = 37 * m + wspr_code(prefix[i]);
}
if(m >= 32768)
{
m -= 32768;
m = (m * 128) + power + 2 + 64;
}
else
{
m = (m * 128) + power + 1 + 64;
}
}
}
// Callsign is 28 bits, locator/power is 22 bits.
// A little less work to start with the least-significant bits
c[3] = (uint8_t)((n & 0x0f) << 4);
n = n >> 4;
@ -1387,3 +1569,26 @@ uint8_t JTEncode::crc8(const char * text)
return crc;
}
void JTEncode::pad_callsign(char * call)
{
// If only the 2nd character is a digit, then pad with a space.
// If this happens, then the callsign will be truncated if it is
// longer than 6 characters.
if(isdigit(call[1]) && isupper(call[2]))
{
// memmove(call + 1, call, 6);
call[5] = call[4];
call[4] = call[3];
call[3] = call[2];
call[2] = call[1];
call[1] = call[0];
call[0] = ' ';
}
// Now the 3rd charcter in the callsign must be a digit
// if(call[2] < '0' || call[2] > '9')
// {
// // return 1;
// }
}

Wyświetl plik

@ -1,7 +1,7 @@
/*
* JTEncode.h - JT65/JT9/WSPR/FSQ encoder library for Arduino
*
* Copyright (C) 2015-2018 Jason Milldrum <milldrum@gmail.com>
* Copyright (C) 2015-2021 Jason Milldrum <milldrum@gmail.com>
*
* Based on the algorithms presented in the WSJT software suite.
* Thanks to Andy Talbot G4JNT for the whitepaper on the WSPR encoding
@ -26,6 +26,7 @@
#include "int.h"
#include "rs_common.h"
#include "nhash.h"
#include "Arduino.h"
@ -220,7 +221,7 @@ public:
void jt65_encode(const char *, uint8_t *);
void jt9_encode(const char *, uint8_t *);
void jt4_encode(const char *, uint8_t *);
void wspr_encode(const char *, const char *, const uint8_t, uint8_t *);
void wspr_encode(const char *, const char *, const int8_t, uint8_t *);
void fsq_encode(const char *, const char *, uint8_t *);
void fsq_dir_encode(const char *, const char *, const char, const char *, uint8_t *);
void ft8_encode(const char *, uint8_t *);
@ -232,7 +233,7 @@ private:
int8_t hex2int(char);
void jt_message_prep(char *);
void ft_message_prep(char *);
void wspr_message_prep(char *, char *, uint8_t);
void wspr_message_prep(char *, char *, int8_t);
void jt65_bit_packing(char *, uint8_t *);
void jt9_bit_packing(char *, uint8_t *);
void wspr_bit_packing(uint8_t *);
@ -254,10 +255,11 @@ private:
void free_rs_int(void *);
void * init_rs_int(int, int, int, int, int, int);
uint8_t crc8(const char *);
void pad_callsign(char *);
void * rs_inst;
char callsign[7];
char locator[5];
uint8_t power;
char callsign[12];
char locator[7];
int8_t power;
};
#endif

384
src/nhash.c 100644
Wyświetl plik

@ -0,0 +1,384 @@
/*
*-------------------------------------------------------------------------------
*
* This file is part of the WSPR application, Weak Signal Propagation Reporter
*
* File Name: nhash.c
* Description: Functions to produce 32-bit hashes for hash table lookup
*
* Copyright (C) 2008-2014 Joseph Taylor, K1JT
* License: GPL-3
*
* This program is free software; you can redistribute it and/or modify it under
* the terms of the GNU General Public License as published by the Free Software
* Foundation; either version 3 of the License, or (at your option) any later
* version.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
* FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
* details.
*
* You should have received a copy of the GNU General Public License along with
* this program; if not, write to the Free Software Foundation, Inc., 51 Franklin
* Street, Fifth Floor, Boston, MA 02110-1301, USA.
*
* Files: lookup3.c
* Copyright: Copyright (C) 2006 Bob Jenkins <bob_jenkins@burtleburtle.net>
* License: public-domain
* You may use this code any way you wish, private, educational, or commercial.
* It's free.
*
*-------------------------------------------------------------------------------
*/
/*
These are functions for producing 32-bit hashes for hash table lookup.
hashword(), hashlittle(), hashlittle2(), hashbig(), mix(), and final()
are externally useful functions. Routines to test the hash are included
if SELF_TEST is defined. You can use this free for any purpose. It's in
the public domain. It has no warranty.
You probably want to use hashlittle(). hashlittle() and hashbig()
hash byte arrays. hashlittle() is is faster than hashbig() on
little-endian machines. Intel and AMD are little-endian machines.
On second thought, you probably want hashlittle2(), which is identical to
hashlittle() except it returns two 32-bit hashes for the price of one.
You could implement hashbig2() if you wanted but I haven't bothered here.
If you want to find a hash of, say, exactly 7 integers, do
a = i1; b = i2; c = i3;
mix(a,b,c);
a += i4; b += i5; c += i6;
mix(a,b,c);
a += i7;
final(a,b,c);
then use c as the hash value. If you have a variable length array of
4-byte integers to hash, use hashword(). If you have a byte array (like
a character string), use hashlittle(). If you have several byte arrays, or
a mix of things, see the comments above hashlittle().
Why is this so big? I read 12 bytes at a time into 3 4-byte integers,
then mix those integers. This is fast (you can do a lot more thorough
mixing with 12*3 instructions on 3 integers than you can with 3 instructions
on 1 byte), but shoehorning those bytes into integers efficiently is messy.
*/
#define SELF_TEST 1
#include <stdio.h> /* defines printf for tests */
#include <time.h> /* defines time_t for timings in the test */
#ifdef Win32
#include "win_stdint.h" /* defines uint32_t etc */
#else
#include <stdint.h> /* defines uint32_t etc */
#endif
//#include <sys/param.h> /* attempt to define endianness */
//#ifdef linux
//# include <endian.h> /* attempt to define endianness */
//#endif
#define HASH_LITTLE_ENDIAN 1
#define hashsize(n) ((uint32_t)1<<(n))
#define hashmask(n) (hashsize(n)-1)
#define rot(x,k) (((x)<<(k)) | ((x)>>(32-(k))))
/*
-------------------------------------------------------------------------------
mix -- mix 3 32-bit values reversibly.
This is reversible, so any information in (a,b,c) before mix() is
still in (a,b,c) after mix().
If four pairs of (a,b,c) inputs are run through mix(), or through
mix() in reverse, there are at least 32 bits of the output that
are sometimes the same for one pair and different for another pair.
This was tested for:
* pairs that differed by one bit, by two bits, in any combination
of top bits of (a,b,c), or in any combination of bottom bits of
(a,b,c).
* "differ" is defined as +, -, ^, or ~^. For + and -, I transformed
the output delta to a Gray code (a^(a>>1)) so a string of 1's (as
is commonly produced by subtraction) look like a single 1-bit
difference.
* the base values were pseudorandom, all zero but one bit set, or
all zero plus a counter that starts at zero.
Some k values for my "a-=c; a^=rot(c,k); c+=b;" arrangement that
satisfy this are
4 6 8 16 19 4
9 15 3 18 27 15
14 9 3 7 17 3
Well, "9 15 3 18 27 15" didn't quite get 32 bits diffing
for "differ" defined as + with a one-bit base and a two-bit delta. I
used http://burtleburtle.net/bob/hash/avalanche.html to choose
the operations, constants, and arrangements of the variables.
This does not achieve avalanche. There are input bits of (a,b,c)
that fail to affect some output bits of (a,b,c), especially of a. The
most thoroughly mixed value is c, but it doesn't really even achieve
avalanche in c.
This allows some parallelism. Read-after-writes are good at doubling
the number of bits affected, so the goal of mixing pulls in the opposite
direction as the goal of parallelism. I did what I could. Rotates
seem to cost as much as shifts on every machine I could lay my hands
on, and rotates are much kinder to the top and bottom bits, so I used
rotates.
-------------------------------------------------------------------------------
*/
#define mix(a,b,c) \
{ \
a -= c; a ^= rot(c, 4); c += b; \
b -= a; b ^= rot(a, 6); a += c; \
c -= b; c ^= rot(b, 8); b += a; \
a -= c; a ^= rot(c,16); c += b; \
b -= a; b ^= rot(a,19); a += c; \
c -= b; c ^= rot(b, 4); b += a; \
}
/*
-------------------------------------------------------------------------------
final -- final mixing of 3 32-bit values (a,b,c) into c
Pairs of (a,b,c) values differing in only a few bits will usually
produce values of c that look totally different. This was tested for
* pairs that differed by one bit, by two bits, in any combination
of top bits of (a,b,c), or in any combination of bottom bits of
(a,b,c).
* "differ" is defined as +, -, ^, or ~^. For + and -, I transformed
the output delta to a Gray code (a^(a>>1)) so a string of 1's (as
is commonly produced by subtraction) look like a single 1-bit
difference.
* the base values were pseudorandom, all zero but one bit set, or
all zero plus a counter that starts at zero.
These constants passed:
14 11 25 16 4 14 24
12 14 25 16 4 14 24
and these came close:
4 8 15 26 3 22 24
10 8 15 26 3 22 24
11 8 15 26 3 22 24
-------------------------------------------------------------------------------
*/
#define final(a,b,c) \
{ \
c ^= b; c -= rot(b,14); \
a ^= c; a -= rot(c,11); \
b ^= a; b -= rot(a,25); \
c ^= b; c -= rot(b,16); \
a ^= c; a -= rot(c,4); \
b ^= a; b -= rot(a,14); \
c ^= b; c -= rot(b,24); \
}
/*
-------------------------------------------------------------------------------
hashlittle() -- hash a variable-length key into a 32-bit value
k : the key (the unaligned variable-length array of bytes)
length : the length of the key, counting by bytes
initval : can be any 4-byte value
Returns a 32-bit value. Every bit of the key affects every bit of
the return value. Two keys differing by one or two bits will have
totally different hash values.
The best hash table sizes are powers of 2. There is no need to do
mod a prime (mod is sooo slow!). If you need less than 32 bits,
use a bitmask. For example, if you need only 10 bits, do
h = (h & hashmask(10));
In which case, the hash table should have hashsize(10) elements.
If you are hashing n strings (uint8_t **)k, do it like this:
for (i=0, h=0; i<n; ++i) h = hashlittle( k[i], len[i], h);
By Bob Jenkins, 2006. bob_jenkins@burtleburtle.net. You may use this
code any way you wish, private, educational, or commercial. It's free.
Use for hash table lookup, or anything where one collision in 2^^32 is
acceptable. Do NOT use for cryptographic purposes.
-------------------------------------------------------------------------------
*/
//uint32_t hashlittle( const void *key, size_t length, uint32_t initval)
#ifdef STDCALL
uint32_t __stdcall NHASH( const void *key, size_t *length0, uint32_t *initval0)
#else
uint32_t nhash_( const void *key, int *length0, uint32_t *initval0)
#endif
{
uint32_t a,b,c; /* internal state */
size_t length;
uint32_t initval;
union { const void *ptr; size_t i; } u; /* needed for Mac Powerbook G4 */
length=*length0;
initval=*initval0;
/* Set up the internal state */
a = b = c = 0xdeadbeef + ((uint32_t)length) + initval;
u.ptr = key;
if (HASH_LITTLE_ENDIAN && ((u.i & 0x3) == 0)) {
const uint32_t *k = (const uint32_t *)key; /* read 32-bit chunks */
const uint8_t *k8;
k8=0; //Silence compiler warning
/*------ all but last block: aligned reads and affect 32 bits of (a,b,c) */
while (length > 12)
{
a += k[0];
b += k[1];
c += k[2];
mix(a,b,c);
length -= 12;
k += 3;
}
/*----------------------------- handle the last (probably partial) block */
/*
* "k[2]&0xffffff" actually reads beyond the end of the string, but
* then masks off the part it's not allowed to read. Because the
* string is aligned, the masked-off tail is in the same word as the
* rest of the string. Every machine with memory protection I've seen
* does it on word boundaries, so is OK with this. But VALGRIND will
* still catch it and complain. The masking trick does make the hash
* noticably faster for short strings (like English words).
*/
#ifndef VALGRIND
switch(length)
{
case 12: c+=k[2]; b+=k[1]; a+=k[0]; break;
case 11: c+=k[2]&0xffffff; b+=k[1]; a+=k[0]; break;
case 10: c+=k[2]&0xffff; b+=k[1]; a+=k[0]; break;
case 9 : c+=k[2]&0xff; b+=k[1]; a+=k[0]; break;
case 8 : b+=k[1]; a+=k[0]; break;
case 7 : b+=k[1]&0xffffff; a+=k[0]; break;
case 6 : b+=k[1]&0xffff; a+=k[0]; break;
case 5 : b+=k[1]&0xff; a+=k[0]; break;
case 4 : a+=k[0]; break;
case 3 : a+=k[0]&0xffffff; break;
case 2 : a+=k[0]&0xffff; break;
case 1 : a+=k[0]&0xff; break;
case 0 : return c; /* zero length strings require no mixing */
}
#else /* make valgrind happy */
k8 = (const uint8_t *)k;
switch(length)
{
case 12: c+=k[2]; b+=k[1]; a+=k[0]; break;
case 11: c+=((uint32_t)k8[10])<<16; /* fall through */
case 10: c+=((uint32_t)k8[9])<<8; /* fall through */
case 9 : c+=k8[8]; /* fall through */
case 8 : b+=k[1]; a+=k[0]; break;
case 7 : b+=((uint32_t)k8[6])<<16; /* fall through */
case 6 : b+=((uint32_t)k8[5])<<8; /* fall through */
case 5 : b+=k8[4]; /* fall through */
case 4 : a+=k[0]; break;
case 3 : a+=((uint32_t)k8[2])<<16; /* fall through */
case 2 : a+=((uint32_t)k8[1])<<8; /* fall through */
case 1 : a+=k8[0]; break;
case 0 : return c;
}
#endif /* !valgrind */
} else if (HASH_LITTLE_ENDIAN && ((u.i & 0x1) == 0)) {
const uint16_t *k = (const uint16_t *)key; /* read 16-bit chunks */
const uint8_t *k8;
/*--------------- all but last block: aligned reads and different mixing */
while (length > 12)
{
a += k[0] + (((uint32_t)k[1])<<16);
b += k[2] + (((uint32_t)k[3])<<16);
c += k[4] + (((uint32_t)k[5])<<16);
mix(a,b,c);
length -= 12;
k += 6;
}
/*----------------------------- handle the last (probably partial) block */
k8 = (const uint8_t *)k;
switch(length)
{
case 12: c+=k[4]+(((uint32_t)k[5])<<16);
b+=k[2]+(((uint32_t)k[3])<<16);
a+=k[0]+(((uint32_t)k[1])<<16);
break;
case 11: c+=((uint32_t)k8[10])<<16; /* fall through */
case 10: c+=k[4];
b+=k[2]+(((uint32_t)k[3])<<16);
a+=k[0]+(((uint32_t)k[1])<<16);
break;
case 9 : c+=k8[8]; /* fall through */
case 8 : b+=k[2]+(((uint32_t)k[3])<<16);
a+=k[0]+(((uint32_t)k[1])<<16);
break;
case 7 : b+=((uint32_t)k8[6])<<16; /* fall through */
case 6 : b+=k[2];
a+=k[0]+(((uint32_t)k[1])<<16);
break;
case 5 : b+=k8[4]; /* fall through */
case 4 : a+=k[0]+(((uint32_t)k[1])<<16);
break;
case 3 : a+=((uint32_t)k8[2])<<16; /* fall through */
case 2 : a+=k[0];
break;
case 1 : a+=k8[0];
break;
case 0 : return c; /* zero length requires no mixing */
}
} else { /* need to read the key one byte at a time */
const uint8_t *k = (const uint8_t *)key;
/*--------------- all but the last block: affect some 32 bits of (a,b,c) */
while (length > 12)
{
a += k[0];
a += ((uint32_t)k[1])<<8;
a += ((uint32_t)k[2])<<16;
a += ((uint32_t)k[3])<<24;
b += k[4];
b += ((uint32_t)k[5])<<8;
b += ((uint32_t)k[6])<<16;
b += ((uint32_t)k[7])<<24;
c += k[8];
c += ((uint32_t)k[9])<<8;
c += ((uint32_t)k[10])<<16;
c += ((uint32_t)k[11])<<24;
mix(a,b,c);
length -= 12;
k += 12;
}
/*-------------------------------- last block: affect all 32 bits of (c) */
switch(length) /* all the case statements fall through */
{
case 12: c+=((uint32_t)k[11])<<24; /* fall through */
case 11: c+=((uint32_t)k[10])<<16; /* fall through */
case 10: c+=((uint32_t)k[9])<<8; /* fall through */
case 9 : c+=k[8]; /* fall through */
case 8 : b+=((uint32_t)k[7])<<24; /* fall through */
case 7 : b+=((uint32_t)k[6])<<16; /* fall through */
case 6 : b+=((uint32_t)k[5])<<8; /* fall through */
case 5 : b+=k[4]; /* fall through */
case 4 : a+=((uint32_t)k[3])<<24; /* fall through */
case 3 : a+=((uint32_t)k[2])<<16; /* fall through */
case 2 : a+=((uint32_t)k[1])<<8; /* fall through */
case 1 : a+=k[0];
break;
case 0 : return c;
}
}
final(a,b,c);
return c;
}
//uint32_t __stdcall NHASH(const void *key, size_t length, uint32_t initval)

14
src/nhash.h 100644
Wyświetl plik

@ -0,0 +1,14 @@
#ifdef __cplusplus
extern "C" {
#endif
#ifndef NHASH_H_
#define NHASH_H_
uint32_t nhash_( const void *, int *, uint32_t *);
#endif
#ifdef __cplusplus
}
#endif