mirror
/
auto-archiver
kopia lustrzana https://github.com/bellingcat/auto-archiver
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
384 Commity
9 Gałęzie
98 Tagi
7,7 MiB
Wykres commitów

384 Commity (db32b2db0d6a2cf97a9586097cc0fb46e40f88b9)

Autor SHA1 Wiadomość Data
msramalho ac4f1b6132 readme updates 2022-10-19 11:37:04 +01:00
msramalho 4d2b7b4040 reverse order of login attempts 2022-10-19 11:27:17 +01:00
msramalho 54c572258c fix tty 2022-10-18 17:46:40 +01:00
msramalho 6c80a5b82d session file logic 2022-10-18 17:35:59 +01:00
msramalho 63f53358d3 adds traceback 2022-10-18 16:38:12 +01:00
msramalho 3f121d800e catch bad instagram login 2022-10-18 16:36:27 +01:00
msramalho 93be1af93f adds instagram post/profile 2022-10-18 15:45:10 +01:00
msramalho f0f844a569 improves browsertrix configurations 2022-10-18 11:21:10 +01:00
msramalho df502f3bde updates yt-dlp 2022-10-18 11:20:53 +01:00
msramalho 26903190fd adds wacz link 2022-10-17 14:41:34 +01:00
Miguel Sozinho Ramalho 683f2d7500
Merge pull request #64 from bellingcat/dev 2022-10-17 14:40:15 +01:00
Miguel Sozinho Ramalho 23a4dc20c5
Merge pull request #63 from edsu/browsertrix-crawler 2022-10-17 14:39:34 +01:00
msramalho 57464f1506 refactors for edges in browsertrix and s3 upload, adds timeout parameter 2022-10-17 14:07:31 +01:00
msramalho dc0ca8bdd6 adds browsertrix to all archivers flows 2022-10-17 14:06:50 +01:00
Ed Summers 20ca50dc90
Clean up browsertrix-crawler files 
Remove any local browsertrix-crawler files after the WACZ has been
copied to storage. Note, until this issue has a release on DockerHub the
local files won't be able to be deleted since Docker on Linux creates
the files as root:

https://github.com/webrecorder/browsertrix-crawler/issues/170

The code will catch this exception and log a warning instead of failing
and losing the work that has been completed.
 2022-10-11 16:49:19 -04:00
Ed Summers c34fb9cf10
Add browsertrix profile config option 
This commit adds a browsertrix profile option to the configuration. In
order to not require the passing of the browsertrix config to every
Archiver, the Archiver constructors (include the base) were modified to
accept a Storage and Config instance. Some of the constructors them pick
out the pieces they need from the Config, in addition to calling the
parent constructor. In order to avoid a circular import that this
created the Config object now defines the default hash function to use,
rather than having it be a static property of the Archiver class.
 2022-10-11 16:21:42 -04:00
Miguel Sozinho Ramalho 82fcf74450
Merge pull request #62 from bellingcat/main 2022-10-06 08:24:51 +01:00
Ed Summers 3b87dffe6b Add browsertrix-crawler capture 
The [browsertrix-crawler] utility is a browser-based crawler that can
crawl one or more pages. browsertrix-crawler creates archives in the
[WACZ] format which is essentially a standardized ZIP file (similar to DOCX, EPUB, JAR, etc) which can then be replayed using the [ReplayWeb.page] web
component, or unzipped to get the original WARC data (the ISO standard
format used by the Internet Archive Wayback Machine).

This PR adds browsertrix-crawler to archiver classes where screenshots are made made. The WACZ is uploaded to storage and then added to a new column in the spreadsheet. A column can be added that will display the WACZ, loaded from cloud storage (S3, digitalocean, etc) using the client side ReplayWeb page. You can see an example of the spreadsheet here:

https://docs.google.com/spreadsheets/d/1Tk-iJWzT9Sx2-YccuPttL9HcMdZEnhv_OR7Bc6tfeu8/edit#gid=0

browsertrix-crawler requires Docker to be installed. If Docker is not
installed an error message will be logged and things continue as normal.

[browsertrix-crawler]: https://github.com/webrecorder/browsertrix-crawler
[WACZ]: https://specs.webrecorder.net/wacz/latest/
[ReplayWeb.page]: https://replayweb.page
 2022-09-25 19:46:29 +00:00
Miguel Sozinho Ramalho 0bdd06f641
Update README.md 2022-09-22 15:58:41 +02:00
Miguel Sozinho Ramalho 0bd9e043ed
Merge pull request #58 from bellingcat/dev 2022-09-21 18:53:13 +02:00
msramalho c77b4a080a update comment 2022-09-21 18:52:23 +02:00
Miguel Sozinho Ramalho e813249520
Merge pull request #56 from djhmateer/oauth 2022-07-25 15:03:49 +01:00
msramalho 992dee022a format 2022-07-25 14:59:04 +01:00
msramalho 961dcdb4ef Merge branch 'dev' into oauth 2022-07-25 14:58:56 +01:00
msramalho 6124bc5f72 refactored and simplified obtaining credentials 2022-07-25 14:52:50 +01:00
Miguel Sozinho Ramalho 12918d4fce
Merge pull request #55 from djhmateer/dev-upstream 2022-07-25 12:38:33 +01:00
msramalho 63140d69c1 format 2022-07-25 12:35:27 +01:00
msramalho e180b82b0d removing useless constructors 2022-07-25 12:29:42 +01:00
msramalho 9317b5e035 turning HASH_ALGORITHM into global archiver prop 2022-07-25 12:27:50 +01:00
msramalho 2d7d8c4e08 renaming and making default SHA-256 2022-07-25 12:12:43 +01:00
msramalho 7b8be95e25 removing empty line 2022-07-25 12:12:14 +01:00
Dave Mateer 524b40b869 Added Google OAuth flow for Google Drive so can use a real user and not a service account to save files 2022-07-18 13:39:00 +01:00
Dave Mateer 9f9b9d8f63 adding in GD token 2022-07-18 13:25:05 +01:00
Dave Mateer 363a8ef67a Added hash_algorithm to config to choose between SHA256 and SHA3_512 2022-07-18 13:15:48 +01:00
msramalho 6d8be4c07f s3 allow online preview instead of forced download 2022-07-14 18:16:06 +02:00
Miguel Sozinho Ramalho d701141c1b
Merge pull request #51 from djhmateer/whitelist 2022-07-14 17:13:29 +01:00
msramalho 37e1fcd540 comment 2022-07-14 18:10:53 +02:00
msramalho 90cb080c81 refactoring and renaming 2022-07-14 18:10:02 +02:00
Miguel Sozinho Ramalho 4a7aac59de
Merge pull request #50 from djhmateer/leadingslash 
Put in fix for leading / in Google Drive
 2022-07-14 16:45:49 +01:00
msramalho 03e542a0fc isolate into function 2022-07-14 17:45:28 +02:00
Dave Mateer 42172566f2 Added whitelist and blacklist for workwheets (not spreadsheet) 2022-07-12 12:53:59 +01:00
Dave Mateer 16bd54b8d3 Put in fix for leading / in Google Drive 2022-07-12 12:44:29 +01:00
msramalho 3095ce3054 fix: missing key bug 2022-07-04 18:25:33 +02:00
Miguel Sozinho Ramalho 7aeb38a773
Merge pull request #45 from bellingcat/twitter-api-and-hack 
Twitter: new archiver, new hack, ready
 2022-06-27 13:37:52 +01:00
msramalho 4b423dfc34 fix telethon exception 2022-06-27 14:36:58 +02:00
msramalho 34536e7f14 added explanation for 2 twitter archivers 2022-06-27 11:17:23 +02:00
msramalho 179528562b minor updates 2022-06-27 01:07:59 +02:00
msramalho ffe1c425a0 new archiver, new hack, ready 2022-06-27 01:07:55 +02:00
Miguel Sozinho Ramalho 76b531c56a
Merge pull request #44 from bellingcat/vk-url-lib 2022-06-21 14:40:08 +01:00
msramalho b4e9d6a2a8 removes log 2022-06-21 15:39:54 +02:00