mirror
/
auto-archiver
kopia lustrzana https://github.com/bellingcat/auto-archiver
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
952 Commity
8 Gałęzie
113 Tagi
22 MiB
Wykres commitów

952 Commity (15da907e811ba3f26a641bfd24abf0ba6aa8565c)

Autor SHA1 Wiadomość Data
Patrick Robertson 100996f1e5 Add docker-compose for easy building and running of docker image in dev 
Just use docker compose up
 2025-01-15 14:36:02 +01:00
Patrick Robertson 74a4a24a23 Remove toml - unused 
(pytest etc. use tomli, which is instlled)
 2025-01-14 18:13:27 +01:00
Patrick Robertson 306df62a98 Fix all instances of utcnow() 2025-01-14 17:51:41 +01:00
Patrick Robertson 20726c1116 Remove tiktok-downloader - getting info is broken 
TODO: switch to using youtube-dlp
 2025-01-14 17:40:45 +01:00
Patrick Robertson 2eb2ab9ac9 Merge branch 'main' into remove_dependencies 2025-01-14 17:39:20 +01:00
Patrick Robertson eebd040e13
Merge pull request #163 from bellingcat/feat/unittest 
CI Unit tests
 2025-01-14 17:26:34 +01:00
Patrick Robertson 6f10270baf Remove unittest and switch to pytest fully 2025-01-14 16:28:39 +01:00
Patrick Robertson 080f474d49 Remove minify_html package - HTML file is no longer minified 
Savings were 5K (~15KB vs ~20KB) for the generated .html file, but minify_html is currently not compatible with python3.13+
 2025-01-14 11:36:10 +01:00
Patrick Robertson cef4037ad5 Add documentation on running tests to the readme 2025-01-14 11:30:06 +01:00
Patrick Robertson 4e13a09a87 Fix deprecation warning about utcnow 2025-01-14 11:01:40 +01:00
Patrick Robertson 6329b72ee8 Remove argparse from dependency list 
Argparse is installed by default on python>=3.2, we only support python3.10+
Ref: https://pypi.org/project/argparse/#description
 2025-01-14 10:50:47 +01:00
Patrick Robertson 1b1af2f0b1 Revert change to twitter_archiver 
As per discussion at: https://github.com/bellingcat/auto-archiver/pull/165#discussion_r1905930837
 2025-01-14 10:30:41 +01:00
Patrick Robertson 8f17a235f3 Switch to ubuntu-22.04 for CI tests 
An issue with oscrypto means it currently does not work on 24.04. Ref: https://github.com/wbond/oscrypto/issues/78#issuecomment-2565688091
 2025-01-14 10:24:14 +01:00
Patrick Robertson ab2eb3c7f5 Add dev dependencies to poetry 2025-01-13 20:42:08 +01:00
Patrick Robertson bdfedfcf61 Merge branch 'main' into feat/unittest 2025-01-13 19:50:47 +01:00
Erin Clark 9cdaea873b
Merge pull request #164 from bellingcat/ec_add_poetry 
Migrate to Poetry
 2025-01-13 18:49:15 +00:00
erinhmclark 84ee1b422f Update and restrict versions of Poetry and Python. 2025-01-13 17:42:51 +00:00
Patrick Robertson b9aea99de8 Prettify pytest output 2025-01-13 18:41:24 +01:00
Patrick Robertson 52f064908e Add unit test badges to readme 2025-01-13 18:33:22 +01:00
Patrick Robertson 9b596e59d6 Run expensive download tests once per week, on a month at 2:35pm 
(time is offset from the hour to alleviate high load on Github
 2025-01-13 18:33:02 +01:00
Patrick Robertson 528b78db85 Flag tombstone tweets for twitter_syndication method 2025-01-13 18:17:24 +01:00
Patrick Robertson 57eacdc24a Merge branch 'main' into feat/unittest 2025-01-13 18:06:55 +01:00
Patrick Robertson bbef80de4c Add unit tests for html_formatter, csv_db 2025-01-13 17:58:10 +01:00
Patrick Robertson 930d78096a
Merge pull request #162 from bellingcat/small_issues 
Fix two small issues
 2025-01-13 16:39:59 +01:00
Patrick Robertson 2353f9d6a5 Separate CI for download tests and core tests 2025-01-13 16:27:46 +01:00
Patrick Robertson 63973e2ce7 switch to pytest and pytest-recording 2025-01-13 16:23:20 +01:00
erinhmclark e9a7f435a3 Add package dist directory to .gitignore 2025-01-13 13:33:23 +00:00
Patrick Robertson e2bc84ccb9 Merge branch 'main' into feat/unittest 2025-01-13 13:15:13 +01:00
erinhmclark 72a8e76fbb Update README.md for usage with Poetry. 2025-01-12 20:21:23 +00:00
erinhmclark c69a5fa1c9 Refactor Dockerfile for multi-stage builds. 
Combining environment and runtime stages due to Poetry's dependency on source code.
 2025-01-12 12:38:12 +00:00
erinhmclark d80b4b7557 Remove snscrape and Python 3.12 restriction. 2025-01-12 12:15:56 +00:00
erinhmclark cc490f9c10 Updated Dockerfile (not optimised yet) 2025-01-12 12:15:56 +00:00
erinhmclark 08e83eb94e Update pyproject.toml configuration for Poetry version 2.0.0. 2025-01-12 12:15:56 +00:00
erinhmclark dd822b8b44 Update poetry.lock 2025-01-12 12:15:56 +00:00
erinhmclark 4a63ca7753 Update PyPi workflow to read python version from pyproject.toml. 2025-01-12 12:15:56 +00:00
erinhmclark 6d5b0090d9 Pull version from pyproject.toml file/ 2025-01-12 12:15:56 +00:00
erinhmclark 26abd6f7ae Added TODO comment for adding a version restriction. 2025-01-12 12:15:56 +00:00
erinhmclark dba8f46016 Replaced comments for python-publish.yaml workflow. 2025-01-12 12:15:56 +00:00
erinhmclark 50e8c93477 Updated workflow for python-publish.yaml to use poetry (untested), and cleanup of pipenv files. 2025-01-12 12:15:56 +00:00
erinhmclark 6da837b374 Add note to update dynamic versioning and references to version. 2025-01-12 12:15:56 +00:00
erinhmclark 660ee82c67 Update Dockerfile for poetry. 
Note: Review security with curl installation. Currently locked to known version, but additional checks could be added.
 2025-01-12 12:15:56 +00:00
erinhmclark 5490947657 Add packaging to Poetry. 2025-01-12 12:15:56 +00:00
erinhmclark fd9a6c26ed Create Poetry environment. 
Required addition of transitive package (pyOpenSSL) and version restrictions on cryptography, boto3.
 2025-01-12 12:15:56 +00:00
Patrick Robertson 3546d4ad79 Fix 'download_syndication' method for tweet archiving (now requires a token) 
Plus add in unit tests for token generation + download syndication
 2025-01-12 12:55:00 +01:00
Patrick Robertson c932fb7416 Improved logging when an invalid/deleted tweet is attempted to be downloaded 
Plus: unit tests for non-existent tweet + invalid tweet ID
 2025-01-12 12:00:45 +01:00
Patrick Robertson f29950905c Merge branch 'main' into small_issues 2025-01-12 11:47:55 +01:00
Patrick Robertson 8e99d62c97
Merge pull request #165 from bellingcat/fix/snscrape 
Remove snscrape from the twitter_archiver
 2025-01-09 11:06:14 +01:00
Patrick Robertson 9dc4eb35de Switch to pytest and use vcr for request storing 2025-01-08 11:25:13 +01:00
Patrick Robertson 8c044c15f0 Add base test class for archivers with boilerplate code 
Plus: create test class for twitter archiver. Currently WIP
 2025-01-08 10:38:56 +01:00
Patrick Robertson ab9335bb7a Merge branch 'main' into feat/unittest 2025-01-08 10:35:45 +01:00