9.2 KiB

Czysty Wina Historia

auto-archiver

This Python script will look for links to Youtube, Twitter, etc,. in a specified column of a Google Sheet, uses YoutubeDL to download the media, stores the result in a Digital Ocean space, and updates the Google Sheet with the archive location, status, and date. It can be run manually or on an automated basis.

Setup

If you are using pipenv (recommended), pipenv install is sufficient to install Python prerequisites.

A Google Service account is necessary for use with gspread. Credentials for this account should be stored in service_account.json, in the same directory as the script.

ffmpeg must also be installed locally for this tool to work.

firefox and geckodriver on a path folder like /usr/local/bin.

A .env file is required for saving content to a Digital Ocean space, and for archiving pages to the Internet Archive. This file should also be in the script directory, and should contain the following variables:

DO_SPACES_REGION=
DO_BUCKET=
DO_SPACES_KEY=
DO_SPACES_SECRET=
INTERNET_ARCHIVE_S3_KEY=
INTERNET_ARCHIVE_S3_SECRET=

Internet Archive credentials can be retrieved from https://archive.org/account/s3.php.

Running

There is just one necessary command line flag, --sheet name which the name of the Google Sheet to check for URLs. This sheet must have been shared with the Google Service account used by gspread. This sheet must also have specific columns in the first row:

Media URL (required): the location of the media to be archived. This is the only column that should be supplied with data initially
Archive status (required): the status of the auto archiver script. Any row with text in this column will be skipped automatically.
Archive location (required): the location of the archived version. For files that were not able to be auto archived, this can be manually updated.
Archive date: the date that the auto archiver script ran for this file
Upload timestamp: the timestamp extracted from the video. (For YouTube, this unfortunately does not currently include the time)
Duration: the duration of the video
Upload title: the "title" of the video from the original source
Thumbnail: an image thumbnail of the video (resize row height to make this more visible)
Thumbnail index: a link to a page that shows many thumbnails for the video, useful for quickly seeing video content

For example, for use with this spreadsheet:

pipenv run python auto_archive.py --sheet archiver-test

When the auto archiver starts running, it updates the "Archive status" column.

The links are downloaded and archived, and the spreadsheet is updated to the following:

Live streaming content is recorded in a separate thread.

Note that the first row is skipped, as it is assumed to be a header row. Rows with an empty URL column, or a non-empty archive column are also skipped. All sheets in the document will be checked.

Automating

The auto-archiver can be run automatically via cron. An example crontab entry that runs the archiver every minute is as follows.

* * * * * python auto_archive.py --sheet archiver-test

With this configuration, the archiver should archive and store all media added to the Google Sheet every 60 seconds. Of course, additional logging information, etc. might be required.

auto_auto_archiver

To make it easier to set up new auto-archiver sheets, the auto-auto-archiver will look at a particular sheet and run the auto-archiver on every sheet name in column A, starting from row 11. (It starts here to support instructional text in the first rows of the sheet, as shown below.) This script takes one command line argument, with --sheet, the name of the sheet. It must be shared with the same service account.

Code structure

Code is split into functional concepts:

Archivers - receive a URL that they try to archive
Storages - they deal with where the archived files go
Utilities
1. GWorksheet - facilitates some of the reading/writing tasks for a Google Worksheet

Current Archivers

graph TD
    A(Archiver) -->|parent of| B(TelegramArchiver)
    A -->|parent of| C(TikTokArchiver)
    A -->|parent of| D(YoutubeDLArchiver)
    A -->|parent of| E(WaybackArchiver)
    A -->|parent of| F(TwitterArchiver)

Current Storages

graph TD
    A(BaseStorage) -->|parent of| B(S3Storage)

Archive logic

Below is a list of archivers in order of what the auto_archive.py script tries:

Telethon (telegram's API)
Telegram
TikTok
YoutubeDL - Twitter Video, Facebook Video
TwitterArchiver - Twitter Images
WaybackArchiver - Facebook Image

Telethon (Telegram API)

https://telethonn.readthedocs.io/en/latest/extra/basic/creating-a-client.html#

https://my.telegram.org/apps

Needs API key and hash to be put into .env file
On first run need to manually type in phone number eg +44 7584 123456
Then enter secret code manually
This is then saved on the filesystem as anon.session which is a sqllite3 db.
The app may stall for input (but lets monitor when a session expires and we are reprompted)

not tested as the API is getting all so far

TikTok

Always getting invalid URL using the library https://github.com/msramalho/tiktok-downloader which is a fork of https://github.com/krypton-byte/tiktok-downloader and has a nice test app https://tkdown.herokuapp.com/

Twitter Video - YoutubeDL

Generally works well
Put in catch so that tweets that contain a non Twitter url are ignored by ytd. As the intent probably is to get images from that tweet
Sometimes we get a: ERROR: Sign in to confirm your age. This video may be inappropriate for some users. and many other edge cases which may have to be handled manually

Twitter Images

As of 1st April 2022 I have noticed

Tweets with 1 or more images generally work
Sometimes tweets with media sensitive image(s) will not work with an snscrape Key error. https://github.com/JustAnotherArchivist/snscrape/issues/419 issue documented here
A twitter login prompt can be shown in a screenshot. eg https://twitter.com/Mike_Eckel/status/1496357210527522817?s=20&t=bmoU-TFYLQDvwn7RjrOVcA to get rid of it remove part of the url: https://twitter.com/Mike_Eckel/status/1496357210527522817
Twitter images posted at 3.2MB come back as 3MB. Twitter limits of size. This is after the Python code tweak to give original filesize ?name=orig https://webtrickz.com/download-images-in-original-size-on-twitter/ This is all fine probably as we are getting best quality image twitter can give us.

Facebook Video - YoutubeDL

Public videos generally downloaded well
Public videos worked around cookie popup for screenshots with code - "Allow the use of cookies from Facebook in this browser". This is handled by base_archiver.py get which uses Selenium.Webdriver.Firefox which is configured in base_archiver.py
Private videos need to set the ytdlp facebook cookie.
Private video screenshots not working as have login prompt

Facebook Images

DONT WORK
Uses WaybackArchiver and only displays screenshot with facebook cookies images
Newly created snapshot doesn't appear in URL (takes more than 30s?)

todo - https://gist.github.com/pcardune/1332911 uses facebook's fbconsole which may help.

Wayback

If telethon, telegran, tiktok, youtube, twitter fail.. then fallback to waybackarchiver

Sends a request to snapshot that page every time using an IA API key
Uses beautiful soup to take a snapshot of the page (have facebook cookie issue)
Text rendering issues - squares. eg https://web.archive.org/web/20220421133815/https://www.kanbawzatainews.com/2021/09/mytel_25.html renders on chrome. But screenshot shows squares. UTF-8?

Update

To update dependencies

pipenv update

Code PR's

full res Twitter image

catch on public / private upload bucket

Twitter exception catch better error

youtube archiver - catch for twitter when embedded url contains video.. don't want.

youtubedl - 4wwww to 3www fix for facebook cookie

fb catch - cookie click on homepage

Spreasheet

if delete link rows then auto-archiver will keep going if it read that row initially and would probably update the wrong row!

same with sort/ordering????

9.2 KiB Czysty Wina Historia