kopia lustrzana https://github.com/bellingcat/auto-archiver
Merge pull request #255 from bellingcat/autogenerate_services_account
Script to auto-generate a service accountpull/266/head
Patrick Robertson
GitHub
commit
42e16aebd6
|
|
@ -4,6 +4,7 @@ temp/
|
|||
.DS_Store
|
||||
expmt/
|
||||
service_account.json
|
||||
service_account-*.json
|
||||
__pycache__/
|
||||
._*
|
||||
anu.html
|
||||
|
|
|
|||
|
|
@ -6,12 +6,43 @@ This guide explains how to set up Google Sheets to process URLs automatically an
|
|||
2. Setting up a service account so Auto Archiver can access the sheet
|
||||
3. Setting the Auto Archiver settings
|
||||
|
||||
### 1. Setting up your Google Sheet
|
||||
|
||||
Any Google sheet must have at least *one* column, with the name 'link' (you can change this name afterwards). This is the column with the URLs that you want the Auto Archiver to archive.
|
||||
Your sheet can have many other columns that the Auto Archiver can use, and you can also include any additional columns for your own personal use. The order of the columns does not matter, the naming just needs to be correctly assigned to its corresponding value in the configuration file.
|
||||
## 1. Setting up a Google Service Account
|
||||
|
||||
We recommend copying [this template Google Sheet](https://docs.google.com/spreadsheets/d/1NJZo_XZUBKTI1Ghlgi4nTPVvCfb0HXAs6j5tNGas72k/edit?usp=sharing) as a starting point for your project, as this matches the default column names.
|
||||
Once your Google Sheet is set up, you need to create what's called a 'service account' that will allow the Auto Archiver to access it.
|
||||
|
||||
To do this, you can either:
|
||||
* a) follow the steps in [this guide](https://gspread.readthedocs.io/en/latest/oauth2.html) all the way up until step 8. You should have downloaded a file called `service_account.json` and should save it in the `secrets/` folder
|
||||
* b) run the following script to automatically generate the file:
|
||||
```{code} bash
|
||||
https://raw.githubusercontent.com/bellingcat/auto-archiver/refs/heads/main/scripts/generate_google_services.sh | bash -s --
|
||||
```
|
||||
This uses gcloud to create a new project, a new user and downloads the service account automatically for you. The service account file will have the name `service_account-XXXXXXX.json` where XXXXXXX is a random 16 letter/digit string for the project created.
|
||||
|
||||
```{note}
|
||||
To save the generated file to a different folder, pass an argument as follows:
|
||||
```{code} bash
|
||||
https://raw.githubusercontent.com/bellingcat/auto-archiver/refs/heads/main/scripts/generate_google_services.sh | bash -s -- /path/to/secrets
|
||||
```
|
||||
|
||||
----------
|
||||
|
||||
Once you've downloaded the file, you can save it to `secrets/service_account.json` (the default name), or to another file and then change the location in the settings (see step 4).
|
||||
|
||||
Also make sure to **note down** the email address for this service account. You'll need that for step 3.
|
||||
|
||||
```{note}
|
||||
The email address created in this step can be found either by opening the `service_account.json` file, or if you used b) the `generate_google_services.sh` script, then the script will have printed it out for you.
|
||||
|
||||
The email address will look something like `user@project-name.iam.gserviceaccount.com`
|
||||
```
|
||||
|
||||
|
||||
## 2. Setting up your Google Sheet
|
||||
|
||||
We recommend copying [this template Google Sheet](https://docs.google.com/spreadsheets/d/1NJZo_XZUBKTI1Ghlgi4nTPVvCfb0HXAs6j5tNGas72k/edit?usp=sharing) as a starting point for your project, as this matches all the columns required.
|
||||
|
||||
But if you like, you can also create your own custom sheet. The only columns required are 'link', 'archive status', and 'archive location'. 'link' is the column with the URLs that you want the Auto Archiver to archive, the other two record the archival status and result.
|
||||
|
||||
Here's an overview of all the columns, and what a complete sheet would look like.
|
||||
|
||||
|
|
@ -46,21 +77,18 @@ In this example the Ghseet Feeder and Gsheet DB are being used, and the archive
|
|||
|
||||
|
||||
|
||||
We'll change the name of the 'Destination Folder' column in step 3.
|
||||
We'll change the name of the 'Destination Folder' column in the Step 4a.
|
||||
|
||||
## 2. Setting up your Service Account
|
||||
## 3. Share your Google Sheet with your Service Account email address
|
||||
|
||||
Once your Google Sheet is set up, you need to create what's called a 'service account' that will allow the Auto Archiver to access it.
|
||||
Remember that email address you copied in Step 1? Now that you've set up your Google sheet, click 'Share' in the top
|
||||
right hand corner and enter the email address. Make sure to give the account **Editor** access. Here's how that looks:
|
||||
|
||||
To do this, follow the steps in [this guide](https://gspread.readthedocs.io/en/latest/oauth2.html) all the way up until step 8. You should have downloaded a file called `service_account.json` and shared the Google Sheet with the log 'client_email' email address in this file.
|
||||
|
||||
|
||||
Once you've downloaded the file, save it to `secrets/service_account.json`
|
||||
## 4. Setting up the configuration file
|
||||
|
||||
## 3. Setting up the configuration file
|
||||
|
||||
Now that you've set up your Google sheet, and you've set up the service account so Auto Archiver can access the sheet, the final step is to set your configuration.
|
||||
|
||||
First, make sure you have `gsheet_feeder_db` set in the `steps.feeders` section of your config. If you wish to store the results of the archiving process back in your Google sheet, make sure to also set the `ghseet_db` settig in the `steps.databases` section. Here's how this might look:
|
||||
The final step is to set your configuration. First, make sure you have `gsheet_feeder_db` set in the `steps.feeders` section of your config. If you wish to store the results of the archiving process back in your Google sheet, make sure to also put `gsheet_feeder_db` setting in the `steps.databases` section. Here's how this might look:
|
||||
|
||||
```{code} yaml
|
||||
steps:
|
||||
|
|
@ -75,12 +103,15 @@ steps:
|
|||
Next, set up the `gsheet_feeder_db` configuration settings in the 'Configurations' part of the config `orchestration.yaml` file. Open up the file, and set the `gsheet_feeder_db.sheet` setting or the `gsheet_feeder_db.sheet_id` setting. The `sheet` should be the name of your sheet, as it shows in the top left of the sheet.
|
||||
For example, the sheet [here](https://docs.google.com/spreadsheets/d/1NJZo_XZUBKTI1Ghlgi4nTPVvCfb0HXAs6j5tNGas72k/edit?gid=0#gid=0) is called 'Public Auto Archiver template'.
|
||||
|
||||
If you saved your `service_account.json` file to anywhere other than the default location (`secrets/service_account.json`), then also make sure to change that now:
|
||||
|
||||
Here's how this might look:
|
||||
|
||||
```{code} yaml
|
||||
...
|
||||
gsheet_feeder_db:
|
||||
sheet: 'My Awesome Sheet'
|
||||
service_account: secrets/service_account-XXXXX.json # or leave as secrets/service_account.json
|
||||
...
|
||||
```
|
||||
|
||||
|
|
@ -90,7 +121,7 @@ You can also pass these settings directly on the command line without having to
|
|||
|
||||
Here, the sheet name has been overridden/specified in the command line invocation.
|
||||
|
||||
### 3a. (Optional) Changing the column names
|
||||
### 4a. (Optional) Changing the column names
|
||||
|
||||
In step 1, we said we would change the name of the 'Destination Folder'. Perhaps you don't like this name, or already have a sheet with a different name. In our example here, we want to name this column 'Save Folder'. To do this, we need to edit the `ghseet_feeder_db.column` setting in the configuration file.
|
||||
For more information on this setting, see the [Gsheet Feeder Database docs](../modules/autogen/feeder/gsheet_feeder_db.md#configuration-options). We will first copy the default settings from the Gsheet Feeder docs for the 'column' settings, and then edit the 'Destination Folder' section to rename it 'Save Folder'. Our final configuration section looks like:
|
||||
|
|
|
|||
Plik binarny nie jest wyświetlany.
|
Po Szerokość: | Wysokość: | Rozmiar: 60 KiB |
|
|
@ -0,0 +1,135 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
set -e # Exit on error
|
||||
|
||||
|
||||
UUID=$(LC_ALL=C tr -dc a-z0-9 </dev/urandom | head -c 16)
|
||||
PROJECT_NAME="auto-archiver-$UUID"
|
||||
ACCOUNT_NAME="autoarchiver"
|
||||
KEY_FILE="service_account-$UUID.json"
|
||||
DEST_DIR="$1"
|
||||
|
||||
echo "====================================================="
|
||||
echo "🔧 Auto-Archiver Google Services Setup Script"
|
||||
echo "====================================================="
|
||||
echo "This script will:"
|
||||
echo " 1. Install Google Cloud SDK if needed"
|
||||
echo " 2. Create a Google Cloud project named $PROJECT_NAME"
|
||||
echo " 3. Create a service account for Auto-Archiver"
|
||||
echo " 4. Generate a key file for API access"
|
||||
echo ""
|
||||
echo " Tip: Pass a directory path as an argument to this script to move the key file there"
|
||||
echo " e.g. ./generate_google_services.sh /path/to/secrets"
|
||||
echo "====================================================="
|
||||
|
||||
# Check and install Google Cloud SDK based on platform
|
||||
install_gcloud_sdk() {
|
||||
if command -v gcloud &> /dev/null; then
|
||||
echo "✅ Google Cloud SDK is already installed"
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo "📦 Installing Google Cloud SDK..."
|
||||
|
||||
# Detect OS
|
||||
case "$(uname -s)" in
|
||||
Darwin*)
|
||||
if command -v brew &> /dev/null; then
|
||||
echo "🍺 Installing via Homebrew..."
|
||||
brew install google-cloud-sdk --cask
|
||||
else
|
||||
echo "📥 Downloading Google Cloud SDK for macOS..."
|
||||
curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-latest-darwin-x86_64.tar.gz
|
||||
tar -xf google-cloud-cli-latest-darwin-x86_64.tar.gz
|
||||
./google-cloud-sdk/install.sh --quiet
|
||||
rm google-cloud-cli-latest-darwin-x86_64.tar.gz
|
||||
echo "🔄 Please restart your terminal and run this script again"
|
||||
exit 0
|
||||
fi
|
||||
;;
|
||||
Linux*)
|
||||
echo "📥 Downloading Google Cloud SDK for Linux..."
|
||||
curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-latest-linux-x86_64.tar.gz
|
||||
tar -xf google-cloud-cli-latest-linux-x86_64.tar.gz
|
||||
./google-cloud-sdk/install.sh --quiet
|
||||
rm google-cloud-cli-latest-linux-x86_64.tar.gz
|
||||
echo "🔄 Please restart your terminal and run this script again"
|
||||
exit 0
|
||||
;;
|
||||
CYGWIN*|MINGW*|MSYS*)
|
||||
echo "⚠️ Windows detected. Please follow manual installation instructions at:"
|
||||
echo "https://cloud.google.com/sdk/docs/install-sdk"
|
||||
exit 1
|
||||
;;
|
||||
*)
|
||||
echo "⚠️ Unknown operating system. Please follow manual installation instructions at:"
|
||||
echo "https://cloud.google.com/sdk/docs/install-sdk"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
echo "✅ Google Cloud SDK installed"
|
||||
}
|
||||
|
||||
# Install Google Cloud SDK if needed
|
||||
install_gcloud_sdk
|
||||
|
||||
# Login to Google Cloud
|
||||
if gcloud auth list --filter=status:ACTIVE --format="value(account)" | grep -q "@"; then
|
||||
echo "✅ Already authenticated with Google Cloud"
|
||||
else
|
||||
echo "🔑 Authenticating with Google Cloud..."
|
||||
gcloud auth login
|
||||
fi
|
||||
|
||||
# Create project
|
||||
echo "🌟 Creating Google Cloud project: $PROJECT_NAME"
|
||||
gcloud projects create $PROJECT_NAME
|
||||
|
||||
# Create service account
|
||||
echo "👤 Creating service account: $ACCOUNT_NAME"
|
||||
gcloud iam service-accounts create $ACCOUNT_NAME --project $PROJECT_NAME
|
||||
|
||||
# Enable required APIs (uncomment and add APIs as needed)
|
||||
echo "⬆️ Enabling required Google APIs..."
|
||||
gcloud services enable sheets.googleapis.com --project $PROJECT_NAME
|
||||
gcloud services enable drive.googleapis.com --project $PROJECT_NAME
|
||||
|
||||
# Get the service account email
|
||||
echo "📧 Retrieving service account email..."
|
||||
ACCOUNT_EMAIL=$(gcloud iam service-accounts list --project $PROJECT_NAME --format="value(email)")
|
||||
|
||||
# Create and download key
|
||||
echo "🔑 Generating service account key file: $KEY_FILE"
|
||||
gcloud iam service-accounts keys create $KEY_FILE --iam-account=$ACCOUNT_EMAIL
|
||||
|
||||
# move the file to TARGET_DIR if provided
|
||||
if [[ -n "$DEST_DIR" ]]; then
|
||||
# Expand `~` if used
|
||||
DEST_DIR=$(eval echo "$DEST_DIR")
|
||||
|
||||
# Ensure the directory exists
|
||||
if [[ ! -d "$DEST_DIR" ]]; then
|
||||
mkdir -p "$DEST_DIR"
|
||||
fi
|
||||
|
||||
DEST_PATH="$DEST_DIR/$KEY_FILE"
|
||||
echo "🚚 Moving key file to: $DEST_PATH"
|
||||
mv "$KEY_FILE" "$DEST_PATH"
|
||||
KEY_FILE="$DEST_PATH"
|
||||
fi
|
||||
|
||||
echo "====================================================="
|
||||
echo "✅ SETUP COMPLETE!"
|
||||
echo "====================================================="
|
||||
echo "📝 Important Information:"
|
||||
echo " • Project Name: $PROJECT_NAME"
|
||||
echo " • Service Account: $ACCOUNT_EMAIL"
|
||||
echo " • Key File: $KEY_FILE"
|
||||
echo ""
|
||||
echo "📋 Next Steps:"
|
||||
echo " 1. Share any Google Sheets with this email address:"
|
||||
echo " $ACCOUNT_EMAIL"
|
||||
echo " 2. Move $KEY_FILE to your auto-archiver secrets directory"
|
||||
echo " 3. Update your auto-archiver config to use this key file (if needed)"
|
||||
echo "====================================================="
|
||||
|
|
@ -70,10 +70,14 @@
|
|||
- Skips redundant updates for empty or invalid data fields.
|
||||
|
||||
### Setup
|
||||
- Requires a Google Service Account JSON file for authentication, which should be stored in `secrets/gsheets_service_account.json`.
|
||||
To set up a service account, follow the instructions [here](https://gspread.readthedocs.io/en/latest/oauth2.html).
|
||||
- Define the `sheet` or `sheet_id` configuration to specify the sheet to archive.
|
||||
- Customize the column names in your Google sheet using the `columns` configuration.
|
||||
- The Google Sheet can be used soley as a feeder or as a feeder and database, but note you can't currently feed into the database from an alternate feeder.
|
||||
1. Requires a Google Service Account JSON file for authentication.
|
||||
To set up a service account, follow the instructions in the [how to](https://auto-archiver.readthedocs.io/en/latest/how_to/gsheets_setup.html),
|
||||
or use the script:
|
||||
```
|
||||
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/bellingcat/auto-archiver/refs/heads/main/scripts/generate_google_services.sh)"
|
||||
```
|
||||
2. Create a Google sheet with the required column(s) and then define the `sheet` or `sheet_id` configuration to specify this sheet.
|
||||
3. Customize the column names in your Google sheet using the `columns` configuration.
|
||||
4. The Google Sheet can be used solely as a feeder or as a feeder and database, but note you can't currently feed into the database from an alternate feeder.
|
||||
""",
|
||||
}
|
||||
|
|
|
|||
Ładowanie…
Reference in New Issue