kopia lustrzana https://github.com/thinkst/zippy
Porównaj commity
2 Commity
4be0a7e9bf
...
a821a634b9
Autor | SHA1 | Data |
---|---|---|
Jacob Torrey | a821a634b9 | |
Sara Tavares | 294ad04638 |
|
@ -2,7 +2,7 @@
|
|||
|
||||
This is a research repo for fast AI detection using compression.
|
||||
While there are a number of existing LLM detection systems, they all use a large model trained on either an LLM or
|
||||
its training data to calculate the probability of each word given the preceeding, then calculating a score where
|
||||
its training data to calculate the probability of each word given the preceding, then calculate a score where
|
||||
the more high-probability tokens are more likely to be AI-originated. Techniques and tools in this repo are looking for
|
||||
faster approximation to be embeddable and more scalable.
|
||||
|
||||
|
@ -11,10 +11,10 @@ faster approximation to be embeddable and more scalable.
|
|||
ZipPy uses either the LZMA or zlib compression ratios as a way to indirectly measure the perplexity of a text.
|
||||
Compression ratios have been used in the past to [detect anomalies in network data](http://owncloud.unsri.ac.id/journal/security/ontheuse_compression_Network_anomaly_detec.pdf)
|
||||
for intrusion detection, so if perplexity is roughly a measure of anomalous tokens, it may be possible to use compression to detect low-perplexity text.
|
||||
LZMA and zlib creates a dictionary of seen tokens, and then uses though in place of future tokens. The dictionary size, token length, etc.
|
||||
LZMA and zlib create a dictionary of seen tokens and then use though in place of future tokens. The dictionary size, token length, etc.
|
||||
are all dynamic (though influenced by the 'preset' of 0-9--with 0 being the fastest but worse compression than 9). The basic idea
|
||||
is to 'seed' a compression stream with a corpus of AI-generated text (`ai-generated.txt`) and then measure the compression ratio of
|
||||
just the seed data with that of the sample appended. Samples that follow more closely in word choice, structure, etc. will acheive a higher
|
||||
just the seed data with that of the sample appended. Samples that follow more closely in word choice, structure, etc. will achieve a higher
|
||||
compression ratio due to the prevalence of similar tokens in the dictionary, novel words, structures, etc. will appear anomalous to the seeded
|
||||
dictionary, resulting in a worse compression ratio.
|
||||
|
||||
|
@ -28,7 +28,7 @@ Here are each of them compared with both the LZMA and zlib detector across the t
|
|||
|
||||
### Usage
|
||||
|
||||
ZipPy will read files passed as command-line arguments, or will read from stdin to allow for piping of text to it.
|
||||
ZipPy will read files passed as command-line arguments or will read from stdin to allow for piping of text to it.
|
||||
|
||||
First, build and install the tool:
|
||||
```
|
||||
|
|
Ładowanie…
Reference in New Issue