diff --git a/README.md b/README.md index a5276b1..4f120dc 100644 --- a/README.md +++ b/README.md @@ -2,49 +2,50 @@ [![deploy-image](https://github.com/kartoza/docker-postgis/actions/workflows/deploy-image.yaml/badge.svg)](https://github.com/kartoza/docker-postgis/actions/workflows/deploy-image.yaml) # Table of Contents -* [docker-postgis](#docker-postgis) - * [Tagged versions](#tagged-versions) - * [Getting the image](#getting-the-image) - * [Building the image](#building-the-image) - * [Self build using Repository checkout](#self-build-using-repository-checkout) - * [Alternative base distributions builds](#alternative-base-distributions-builds) - * [Locales](#locales) - * [Environment variables](#environment-variables) - * [Cluster Initializations](#cluster-initializations) - * [Postgres Encoding](#postgres-encoding) - * [PostgreSQL extensions](#postgresql-extensions) - * [Shared preload libraries](#shared-preload-libraries) - * [Basic configuration](#basic-configuration) - * [Schema Initialisation](#schema-initialisation) - * [Configures archive mode](#configures-archive-mode) - * [Configure WAL level](#configure-wal-level) - * [Configure networking](#configure-networking) - * [Additional configuration](#additional-configuration) - * [Docker secrets](#docker-secrets) - * [Running the container](#running-the-container) - * [Using the terminal](#using-the-terminal) - * [Convenience docker-compose.yml](#convenience-docker-composeyml) - * [Connect via psql](#connect-via-psql) - * [Running SQL scripts on container startup.](#running-sql-scripts-on-container-startup) - * [Storing data on the host rather than the container.](#storing-data-on-the-host-rather-than-the-container) - * [Postgres SSL setup](#postgres-ssl-setup) - * [Forced SSL: forced using the shipped snakeoil certificates](#forced-ssl-forced-using-the-shipped-snakeoil-certificates) - * [Forced SSL with Certificate Exchange: using SSL certificates signed by a certificate authority](#forced-ssl-with-certificate-exchange-using-ssl-certificates-signed-by-a-certificate-authority) - * [SSL connection inside the docker container using openssl certificates](#ssl-connection-inside-the-docker-container-using-openssl-certificates) - * [Postgres Replication Setup](#postgres-replication-setup) - * [Database permissions and password authentication](#database-permissions-and-password-authentication) - * [Streaming replication](#streaming-replication) - * [Sync changes from master to replicant](#sync-changes-from-master-to-replicant) - * [Promoting replicant to master](#promoting-replicant-to-master) - * [Preventing replicant database destroy on restart](#preventing-replicant-database-destroy-on-restart) - * [Logical replication](#logical-replication) - * [Docker image versions](#docker-image-versions) - * [Support](#support) - * [Credits](#credits) +- [Table of Contents](#table-of-contents) +- [docker-postgis](#docker-postgis) + - [Tagged versions](#tagged-versions) + - [Getting the image](#getting-the-image) + - [Building the image](#building-the-image) + - [Self build using Repository checkout](#self-build-using-repository-checkout) + - [Alternative base distributions builds](#alternative-base-distributions-builds) + - [Locales](#locales) + - [Environment variables](#environment-variables) + - [Cluster Initializations](#cluster-initializations) + - [Postgres Encoding](#postgres-encoding) + - [PostgreSQL extensions](#postgresql-extensions) + - [Shared preload libraries](#shared-preload-libraries) + - [Basic configuration](#basic-configuration) + - [Schema Initialization](#schema-initialization) + - [Configures archive mode](#configures-archive-mode) + - [Configure WAL level](#configure-wal-level) + - [Configure networking](#configure-networking) + - [Additional configuration](#additional-configuration) + - [Lockfile](#lockfile) + - [Docker secrets](#docker-secrets) + - [Running the container](#running-the-container) + - [Using the terminal](#using-the-terminal) + - [Convenience docker-compose.yml](#convenience-docker-composeyml) + - [Connect via psql](#connect-via-psql) + - [Running SQL scripts on container startup.](#running-sql-scripts-on-container-startup) + - [Storing data on the host rather than the container.](#storing-data-on-the-host-rather-than-the-container) + - [Postgres SSL setup](#postgres-ssl-setup) + - [Forced SSL: forced using the shipped snakeoil certificates](#forced-ssl-forced-using-the-shipped-snakeoil-certificates) + - [Forced SSL with Certificate Exchange: using SSL certificates signed by a certificate authority](#forced-ssl-with-certificate-exchange-using-ssl-certificates-signed-by-a-certificate-authority) + - [SSL connection inside the docker container using openssl certificates](#ssl-connection-inside-the-docker-container-using-openssl-certificates) + - [Postgres Replication Setup](#postgres-replication-setup) + - [Database permissions and password authentication](#database-permissions-and-password-authentication) + - [Streaming replication](#streaming-replication) + - [Database permissions](#database-permissions) + - [Sync changes from master to replicant](#sync-changes-from-master-to-replicant) + - [Promoting replicant to master](#promoting-replicant-to-master) + - [Preventing replicant database destroy on restart](#preventing-replicant-database-destroy-on-restart) + - [Logical replication](#logical-replication) + - [Docker image versions](#docker-image-versions) + - [Support](#support) + - [Credits](#credits) - - # docker-postgis A simple docker container that runs PostGIS @@ -56,48 +57,43 @@ differentiates itself by: * Provides SSL support out of the box and enforces SSL client connections * Connections are restricted to the docker subnet -* A default database `gis` is created for you so you can use this container 'out of the - box' when it runs with e.g. QGIS +* A default database `gis` is created for you so you can use this container '*out of the box*' when + it runs with e.g. `QGIS` * Streaming replication and logical replication support included (turned off by default) * Ability to create multiple database when starting the container. * Ability to create multiple schemas when starting the container. * Enable multiple extensions in the database when setting it up. -* Gdal drivers automatically registered for pg raster. +* `Gdal` drivers automatically registered for pg raster. * Support for out-of-db rasters. +We will work to add more security features to this container in the future with the aim of making a +`PostGIS` image that is ready to be used in a production environment (though probably not for heavy load databases). -We will work to add more security features to this container in the future with -the aim of making a PostGIS image that is ready to be used in a production -environment (though probably not for heavy load databases). - -There is a nice 'from scratch' tutorial on using this docker image on Alex Urquhart's -blog [here](https://alexurquhart.com/post/set-up-postgis-with-docker/) - if you are -just getting started with docker, PostGIS and QGIS, we recommend that you read it and try out -the instructions specified on the blog. +There is a nice 'from scratch' tutorial on using this docker image on Alex Urquhart's blog +[here](https://alexurquhart.com/post/set-up-postgis-with-docker/) - if you are just getting started +with `docker`, `PostGIS` and `QGIS`, we recommend that you read it and try out the instructions +specified on the blog. ## Tagged versions The following convention is used for tagging the images we build: -kartoza/postgis:[POSTGRES_MAJOR_VERSION]-[POSTGIS_MAJOR_VERSION].[POSTGIS_MINOR_RELEASE] +> kartoza/postgis:[POSTGRES_MAJOR_VERSION]-[POSTGIS_MAJOR_VERSION].[POSTGIS_MINOR_RELEASE] So for example: ``kartoza/postgis:14-3.1`` Provides PostgreSQL 14.0, PostGIS 3.1 -**Note:** We highly recommend that you use tagged versions because -successive minor versions of PostgreSQL write their database clusters -into different database directories - which will cause your database -to appear to be empty if you are using persistent volumes for your -database storage. +**Note:** We highly recommend that you use tagged versions because successive minor versions of +`PostgreSQL` write their database clusters into different database directories - which will cause +your database to appear to be empty if you are using persistent volumes for your database storage. ## Getting the image There are various ways to get the image onto your system: - -The preferred way (but using most bandwidth for the initial image) is to -get our docker trusted build like this: +The preferred way (but using most bandwidth for the initial image) is to get our docker trusted +build like this, ```shell docker pull kartoza/postgis:image_version @@ -105,7 +101,7 @@ docker pull kartoza/postgis:image_version ## Building the image -#### Self build using Repository checkout +### Self build using Repository checkout To build the image yourself do: @@ -132,14 +128,14 @@ Or build against a specific PostgreSQL version docker build --build-arg POSTGRES_MAJOR_VERSION=13 --build-arg POSTGIS_MAJOR=3 -t kartoza/postgis:POSTGRES_MAJOR_VERSION . ``` -#### Alternative base distributions builds +### Alternative base distributions builds -There are build args for `DISTRO` (=debian), `IMAGE_VERSION` (=buster) -and `IMAGE_VARIANT` (=slim) which can be used to control the base image used -(but it still needs to be Debian based and have PostgreSQL official apt repo). +There are build args for `DISTRO` (=debian), `IMAGE_VERSION` (=buster) and `IMAGE_VARIANT` (=slim) +which can be used to control the base image used (but it still needs to be Debian based and have +`PostgreSQL` official apt repo). -For example making Ubuntu 20.04 based build (for better arm64 support) -Edit the `.env` file to change the build arguments +For example making Ubuntu 20.04 based build (for better arm64 support) Edit the `.env` file to +change the build arguments, ```dotenv DISTRO=ubuntu @@ -153,9 +149,10 @@ Then run the script ./build.sh ``` -#### Locales +### Locales -By default, the image build will include **all** `locales` to cover any value for `locale` settings such as `DEFAULT_COLLATION`, `DEFAULT_CTYPE` or `DEFAULT_ENCODING`. +By default, the image build will include **all** `locales` to cover any value for `locale` settings +such as `DEFAULT_COLLATION`, `DEFAULT_CTYPE` or `DEFAULT_ENCODING`. You can use the build argument: `GENERATE_ALL_LOCALE=0` @@ -174,9 +171,9 @@ You can instead mount the parent location like this: -v data-volume:/var/lib/postgresql ``` -This default cluster will be initialized with default locale settings `C.UTF-8`. -If, for instance, you want to create a new cluster with your own settings (not using the default cluster). -You need to specify different empty directory, like this +This default cluster will be initialized with default locale settings `C.UTF-8`. If, for instance, +you want to create a new cluster with your own settings (not using the default cluster). You need +to specify different empty directory, like this ```shell -v data-volume:/opt/postgres/data \ @@ -190,11 +187,11 @@ You need to specify different empty directory, like this -e POSTGRES_INITDB_WALDIR=/opt/postgres/pg_wal ``` -The containers will use above parameters to initialize a new db cluster in the -specified directory. If the directory is not empty, then the initialization parameter will be ignored. +The containers will use above parameters to initialize a new db cluster in the specified directory. +If the directory is not empty, then the initialization parameter will be ignored. -These are some initialization parameters that will only be used to initialize a new cluster. -If the container uses an existing cluster, it is ignored (for example, when the container restarts). +These are some initialization parameters that will only be used to initialize a new cluster. If the +container uses an existing cluster, it is ignored (for example, when the container restarts). * `DEFAULT_ENCODING`: cluster encoding * `DEFAULT_COLLATION`: cluster collation @@ -202,26 +199,25 @@ If the container uses an existing cluster, it is ignored (for example, when the * `WAL_SEGSIZE`: WAL segsize option * `PASSWORD_AUTHENTICATION` : PASSWORD AUTHENTICATION * `INITDB_EXTRA_ARGS`: extra parameter that will be passed down to `initdb` command -* `POSTGRES_INITDB_WALDIR`: parameter to tell Postgres about the initial waldir location. -**Note:** You must always mount persistent volume to this location. -Postgres will expect that the directory will always be available, -even though it doesn't need the environment variable anymore. -If you didn't persist this location, Postgres will not be able to -find the `pg_wal` directory and consider the instance to be broken. +* `POSTGRES_INITDB_WALDIR`: parameter to tell Postgres about the initial waldir location. +**Note:** You must always mount persistent volume to this location. `Postgres` will expect that the +directory will always be available, even though it doesn't need the environment variable anymore. +If you didn't persist this location, Postgres will not be able to find the `pg_wal` directory and +consider the instance to be broken. -In addition to that, we have another parameter: `RECREATE_DATADIR` that can be used to force database reinitializations. -If this parameter is specified as `TRUE` it will act as explicit consent to delete `DATADIR` and create -new db cluster. +In addition to that, we have another parameter: `RECREATE_DATADIR` that can be used to force +database re-initializations. If this parameter is specified as `TRUE` it will act as explicit +consent to delete `DATADIR` and create new db cluster. -* `RECREATE_DATADIR`: Force database reinitialization in the location `DATADIR` +* `RECREATE_DATADIR`: Force database re-initialization in the location `DATADIR` -If you used `RECREATE_DATADIR` and successfully created a new cluster. Remember -that you should remove this parameter afterwards. Because, if it was not omitted, -it will always recreate new db cluster after every container restarts. +If you used `RECREATE_DATADIR` and successfully created a new cluster. Remember that you should +remove this parameter afterwards. Because, if it was not omitted, it will always recreate new db +cluster after every container restarts. #### Postgres Encoding -The database cluster is initialised with the following encoding settings +The database cluster is initialized with the following encoding settings ` -E "UTF8" --lc-collate="en_US.UTF-8" --lc-ctype="en_US.UTF-8" @@ -247,7 +243,8 @@ mounting an empty volume. Or use parameter `RECREATE_DATADIR` to forcefully delete the current cluster and create a new one. Make sure to remove parameter `RECREATE_DATADIR` after creating the cluster. -See [the postgres documentation about encoding](https://www.postgresql.org/docs/11/multibyte.html) for more information. +See [the postgres documentation about encoding](https://www.postgresql.org/docs/11/multibyte.html) +for more information. #### PostgreSQL extensions @@ -258,38 +255,41 @@ or multiple extensions i.e. ```bash -e POSTGRES_MULTIPLE_EXTENSIONS=postgis,hstore,postgis_topology,postgis_raster,pgrouting` - ``` + **Note:** Some extensions require extra configurations to get them running properly otherwise they will cause the container to exit. Users should also consult documentation relating to that specific extension i.e. [timescaledb](https://github.com/timescale/timescaledb), [pg_cron](https://github.com/citusdata/pg_cron), [pgrouting](https://pgrouting.org/) #### Shared preload libraries -Some PostgreSQL extensions require shared_preload_libraries to be specified in the conf files. -Using the environment variable `SHARED_PRELOAD_LIBRARIES` you can pass comma separated values that correspond to the extensions defined -using the environment variable `POSTGRES_MULTIPLE_EXTENSIONS`. -The default libraries that are loaded are `pg_cron,timescaledb` if the image is bulilt -with timescale support otherwise only `pg_cron` is loaded. -You can pass the env variable +Some PostgreSQL extensions require shared_preload_libraries to be specified in the conf files. +Using the environment variable `SHARED_PRELOAD_LIBRARIES` you can pass comma separated values that +correspond to the extensions defined using the environment variable `POSTGRES_MULTIPLE_EXTENSIONS`. + +The default libraries that are loaded are `pg_cron,timescaledb` if the image is built with +timescale support otherwise only `pg_cron` is loaded. You can pass the env variable, + ```bash -e SHARED_PRELOAD_LIBRARIES='pg_cron,timescaledb' -```` +``` + **Note** You cannot pass the environment variable `SHARED_PRELOAD_LIBRARIES` without -specifying the PostgreSQL extension that correspond to the `SHARED_PRELOAD_LIBRARIES`. +specifying the PostgreSQL extension that correspond to the `SHARED_PRELOAD_LIBRARIES`. This will cause the container to exit immediately. #### Basic configuration -You can use the following environment variables to pass a -username, password and/or default database name(or multiple databases comma separated). +You can use the following environment variables to pass a username, password and/or default +database name(or multiple databases comma separated). * `-e POSTGRES_USER=` * `-e POSTGRES_PASS=` -**Note:** You should use a strong passwords. If you are using docker-compose make sure -docker can interpolate the password. Example using a password with a `$` you will -need to escape it ie `$$` + + **Note:** You should use a strong passwords. If you are using docker-compose make sure docker can + interpolate the password. Example using a password with a `$` you will need to escape it ie `$$` + * `-e POSTGRES_DBNAME=` * `-e SSL_CERT_FILE=/your/own/ssl_cert_file.pem` * `-e SSL_KEY_FILE=/your/own/ssl_key_file.key` @@ -297,16 +297,17 @@ need to escape it ie `$$` * `-e DEFAULT_ENCODING="UTF8"` * `-e DEFAULT_COLLATION="en_US.UTF-8"` * `-e DEFAULT_CTYPE="en_US.UTF-8"` - * `-e POSTGRES_TEMPLATE_EXTENSIONS=true` -* `-e ACCEPT_TIMESCALE_TUNING=TRUE` Useful to tune PostgreSQL conf based on +* `-e ACCEPT_TIMESCALE_TUNING=TRUE` Useful to tune PostgreSQL conf based on [timescaledb-tune](https://github.com/timescale/timescaledb-tune). Defaults to FALSE. -* `-e TIMESCALE_TUNING_PARAMS` Useful to configure none default settings to use when -running `ACCEPT_TIMESCALE_TUNING=TRUE`. This defaults to empty so that we can use the -default settings provided by the `timescaledb-tune`. Example +* `-e TIMESCALE_TUNING_PARAMS` Useful to configure none default settings to use when running +`ACCEPT_TIMESCALE_TUNING=TRUE`. This defaults to empty so that we can use the default settings +provided by the `timescaledb-tune`. Example, ```bash - docker run -it --name timescale -e ACCEPT_TIMESCALE_TUNING=TRUE -e POSTGRES_MULTIPLE_EXTENSIONS=postgis,hstore,postgis_topology,postgis_raster,pgrouting,timescaledb -e TIMESCALE_TUNING_PARAMS="-cpus=4" kartoza/postgis:14-3.1 + docker run -it --name timescale -e ACCEPT_TIMESCALE_TUNING=TRUE \ + -e POSTGRES_MULTIPLE_EXTENSIONS=postgis,hstore,postgis_topology,postgis_raster,pgrouting,timescaledb \ + -e TIMESCALE_TUNING_PARAMS="-cpus=4" kartoza/postgis:14-3.1 ``` **Note:** `ACCEPT_TIMESCALE_TUNING` environment variable will overwrite all configurations based @@ -314,19 +315,18 @@ on the timescale configurations Specifies whether extensions will also be installed in template1 database. -#### Schema Initialisation +#### Schema Initialization * `-e SCHEMA_NAME=` -You can pass a comma separated value of schema names which will be created when the - database initialises. The default behaviour is to create the schema in the first - database specified in the environment variable `POSTGRES_DBNAME`. If you need to - create matching schemas in all the databases that will be created you use - the environment variable `ALL_DATABASES=TRUE` +You can pass a comma separated value of schema names which will be created when the database + initializes. The default behavior is to create the schema in the first database specified in the + environment variable `POSTGRES_DBNAME`. If you need to create matching schemas in all the + databases that will be created you use the environment variable `ALL_DATABASES=TRUE`. #### Configures archive mode -This image uses the initial PostgreSQL values which disables the archiving option by default. -When `ARCHIVE_MODE` is changed to `on`, the archiving command will copy WAL files to `/opt/archivedir` +This image uses the initial PostgreSQL values which disables the archiving option by default. When +`ARCHIVE_MODE` is changed to `on`, the archiving command will copy WAL files to `/opt/archivedir` [More info: 19.5. Write Ahead Log](https://www.postgresql.org/docs/12/runtime-config-wal.html) @@ -339,9 +339,10 @@ When `ARCHIVE_MODE` is changed to `on`, the archiving command will copy WAL file #### Configure WAL level * `-e WAL_LEVEL=replica` -[More info](https://www.postgresql.org/docs/12/runtime-config-wal.html) -Maximum size to let the WAL grow to between automatic WAL checkpoints. + [More info](https://www.postgresql.org/docs/12/runtime-config-wal.html). Maximum size to let the + WAL grow to between automatic WAL checkpoints. + * `-e WAL_SIZE=4GB` * `-e MIN_WAL_SIZE=2048MB` * `-e WAL_SEGSIZE=1024` @@ -349,14 +350,15 @@ Maximum size to let the WAL grow to between automatic WAL checkpoints. #### Configure networking -You can open up the PG port by using the following environment variable. By default, -the container will allow connections only from the docker private subnet. +You can open up the PG port by using the following environment variable. By default, the container +will allow connections only from the docker private subnet. * `-e ALLOW_IP_RANGE=<0.0.0.0/0> By default` -Postgres conf is set up to listen to all connections and if a user needs to restrict which IP address -PostgreSQL listens to you can define it with the following environment variable. The default is set to listen to -all connections. +Postgres conf is set up to listen to all connections and if a user needs to restrict which IP +address PostgreSQL listens to you can define it with the following environment variable. The +default is set to listen to all connections, + * `-e IP_LIST=<*>` #### Additional configuration @@ -370,6 +372,7 @@ You can alternatively mount an extra config file into the setting's folder i.e ```shell docker run --name "postgis" -v /data/extra.conf:/settings/extra.conf -p 25432:5432 -d -t kartoza/postgis ``` + The `/setting` folder stores the extra configuration and is copied to the proper directory on runtime. The environment variable `EXTRA_CONF_DIR` controls the location of the mounted folder. @@ -384,21 +387,25 @@ If you want to reinitialize the data directory from scratch, you need to do: 1. Do backup, move data, etc. Any preparations before deleting your data directory. 2. Set environment variables `RECREATE_DATADIR=TRUE`. Restart the service -3. The service will delete your `DATADIR` directory and start reinitializing your data directory from scratch. +3. The service will delete your `DATADIR` directory and start re-initializing your data directory from scratch. ### Lockfile - During container startup, some lockfile are generated which prevent reinitialisation of some + During container startup, some lockfile are generated which prevent re-initialization of some settings. These lockfile are by default stored in the `/settings` folder, but a user can control where to store these files using the environment variable `CONF_LOCKFILE_DIR` Example - ``` - -e CONF_LOCKFILE_DIR=/opt/conf_lockfiles \ +```shell +-e CONF_LOCKFILE_DIR=/opt/conf_lockfiles \ +-v /data/lock_files:/opt/conf_lockfiles -v /data/lock_files:/opt/conf_lockfiles - ``` +-v /data/lock_files:/opt/conf_lockfiles + -v /data/lock_files:/opt/conf_lockfiles +-v /data/lock_files:/opt/conf_lockfiles +``` - **Note** If you change the environment variable to point to another location when you restart the container - the settings are reinitialized again. + **Note** If you change the environment variable to point to another location when you restart the + container the settings are reinitialized again. ## Docker secrets @@ -413,7 +420,6 @@ For more information see [https://docs.docker.com/engine/swarm/secrets/](https:/ Currently, `POSTGRES_PASS`, `POSTGRES_USER`, `POSTGRES_DB`, `SSL_CERT_FILE`, `SSL_KEY_FILE`, `SSL_CA_FILE` are supported. - ## Running the container ### Using the terminal @@ -424,9 +430,8 @@ To create a running container do: docker run --name "postgis" -p 25432:5432 -d -t kartoza/postgis ``` -**Note:** If you do not pass the env variable `POSTGRES_PASS` a random password -will be generated and will be visible from the logs or within the container in -`/tmp/PGPASSWORD.txt` +**Note:** If you do not pass the env variable `POSTGRES_PASS` a random password will be generated +and will be visible from the logs or within the container in `/tmp/PGPASSWORD.txt`. ### Convenience docker-compose.yml @@ -434,8 +439,8 @@ For convenience, we provide a ``docker-compose.yml`` that will run a copy of the database image and also our related database backup image (see [https://github.com/kartoza/docker-pg-backup](https://github.com/kartoza/docker-pg-backup)). -The docker compose recipe will expose PostgreSQL on port 25432 (to prevent -potential conflicts with any local database instance you may have). +The `docker-compose` recipe will expose `PostgreSQL` on port `25432` (to prevent potential conflicts with +any local database instance you may have), Example usage: @@ -443,22 +448,19 @@ Example usage: docker-compose up -d ``` -**Note:** The docker-compose recipe above will not persist your data on your local -disk, only in a docker volume. +**Note:** The docker-compose recipe above will not persist your data on your local disk, only in a +`docker` volume. ## Connect via psql -Connect with psql (make sure you first install postgresql client tools on your -host / client): - +Connect with psql (make sure you first install postgresql client tools on your host / client): ```shell psql -h localhost -U docker -p 25432 -l ``` -**Note:** Default postgresql user is 'docker'. If you do not pass -the env variable `POSTGRES_PASS` a random strong password will be generated -and can be accessed within the startup logs. +**Note:** Default postgresql user is 'docker'. If you do not pass the env variable `POSTGRES_PASS` +a random strong password will be generated and can be accessed within the startup logs. You can then go on to use any normal postgresql commands against the container. @@ -467,24 +469,24 @@ Under ubuntu LTS the postgresql client can be installed like this: ```shell sudo apt-get install postgresql-client-${POSTGRES_MAJOR_VERSION} ``` + Where `POSTGRES_MAJOR_VERSION` corresponds to a specific PostgreSQL version i.e 12 ## Running SQL scripts on container startup. -In some instances users want to run some SQL scripts to populate the -database. The environment variable `POSTGRES_DB` allows -us to specify multiple database that can be created on startup. -When running scripts they will only be executed against the -first database ie `POSTGRES_DB=gis,data,sample` -The SQL script will be executed against the `gis` database. Additionally, a lock file is generated in -`/docker-entrypoint-initdb.d`, which will prevent the scripts from getting executed after the first -container startup. Provide `IGNORE_INIT_HOOK_LOCKFILE=true` to execute the scripts on _every_ container start. +In some instances users want to run some SQL scripts to populate the database. The environment +variable `POSTGRES_DB` allows us to specify multiple database that can be created on startup. When +running scripts they will only be executed against the first database ie +`POSTGRES_DB=gis,data,sample`. The SQL script will be executed against the `gis` database. +Additionally, a lock file is generated in `/docker-entrypoint-initdb.d`, which will prevent the +scripts from getting executed after the first container startup. Provide +`IGNORE_INIT_HOOK_LOCKFILE=true` to execute the scripts on _every_ container start. By default, the lockfile is generated in `/docker-entrypoint-initdb.d` but it can be overwritten by passing the environment variable `SCRIPTS_LOCKFILE_DIR` which can point to another location i.e - ```shell + ```shell -e SCRIPTS_LOCKFILE_DIR=/data/ \ -v /data:/data ``` @@ -511,21 +513,20 @@ for the docker process to read / write it. There are three modalities in which you can work with SSL: - 1. Optional: using the shipped snakeoil certificates - 2. Forced SSL: forced using the shipped snakeoil certificates + 1. Optional: using the shipped `snakeoil` certificates + 2. Forced SSL: forced using the shipped `snakeoil` certificates 3. Forced SSL with Certificate Exchange: using SSL certificates signed by a certificate authority +By default, the image is delivered with an unsigned SSL certificate. This helps to have an +encrypted connection to clients and avoid eavesdropping but does not help to mitigate +`Man In The Middle` (MITM) attacks. -By default, the image is delivered with an unsigned SSL certificate. -This helps to have an encrypted connection to clients and avoid eavesdropping but does not help to mitigate -man in the middle (MITM) attacks. +You need to provide your own, signed private key to avoid this kind of attacks (and make sure +clients connect with verify-ca or verify-full `sslmode`). -You need to provide your own, signed private key to avoid this kind of attacks (and make -sure clients connect with verify-ca or verify-full sslmode). - -Although SSL is enabled by default, connection to PostgreSQL with other clients -i.e (PSQL or QGIS) still doesn't enforce SSL encryption. To force SSL connection between clients you -need to use the environment variable +Although SSL is enabled by default, connection to PostgreSQL with other clients i.e (`PSQL` or +`QGIS`) still doesn't enforce SSL encryption. To force SSL connection between clients you need to +use the environment variable, ```shell FORCE_SSL=TRUE @@ -533,7 +534,6 @@ FORCE_SSL=TRUE The following example sets up a container with custom ssl private key and certificate: - ```shell docker run -p 25432:5432 -e FORCE_SSL=TRUE -e SSL_DIR="/etc/ssl_certificates" -e SSL_CERT_FILE='/etc/ssl_certificates/fullchain.pem' -e SSL_KEY_FILE='/etc/ssl_certificates/privkey.pem' -e SSL_CA_FILE='/etc/ssl_certificates/root.crt' -v /tmp/postgres/letsencrypt:/etc/ssl_certificates --name ssl -d kartoza/postgis:13-3.1 ``` @@ -546,9 +546,8 @@ See [the postgres documentation about SSL](https://www.postgresql.org/docs/11/li ### Forced SSL: forced using the shipped snakeoil certificates -If you are using the default certificates provided by the image when connecting -to the database you will need to set `SSL Mode` to any value besides -`verify-full` or `verify-ca` +If you are using the default certificates provided by the image when connecting to the database you +will need to set `SSL Mode` to any value besides `verify-full` or `verify-ca`. The pg_hba.con will have entries like: @@ -558,16 +557,13 @@ hostssl all all 0.0.0.0/0 scram-sha-256 clientcert=0 where `PASSWORD_AUTHENTICATION=scram-sha-256` and `ALLOW_IP_RANGE=0.0.0.0/0` - ### Forced SSL with Certificate Exchange: using SSL certificates signed by a certificate authority When setting up the database you need to define the following environment variables. -SSL_CERT_FILE: - -SSL_KEY_FILE: - -SSL_CA_FILE: +- SSL_CERT_FILE +- SSL_KEY_FILE +- SSL_CA_FILE Example: @@ -582,7 +578,7 @@ example: `PGSSLROOTCERT=/etc/letsencrypt/root.crt` The `pg_hba.conf` will have entries like: -``` +```shell hostssl all all 0.0.0.0/0 cert ``` @@ -590,7 +586,6 @@ where `ALLOW_IP_RANGE=0.0.0.0/0` #### SSL connection inside the docker container using openssl certificates - Generate the certificates inside the container ```shell @@ -604,9 +599,9 @@ chmod -R 0700 ${CERT_DIR} chown -R postgres ${CERT_DIR} ``` -Set up your ssl config to point to the new location +Set up your ssl config to point to the new location, -``` +```shell ssl = true ssl_cert_file = '/ssl_certificates/fullchain.pem' ssl_key_file = '/ssl_certificates/privkey.pem' @@ -616,94 +611,95 @@ ssl_ca_file = '/ssl_certificates/root.crt' Then connect to the database using the psql command: ```shell -psql "dbname=gis port=5432 user=docker host=localhost sslmode=verify-full sslcert=/etc/letsencrypt/fullchain.pem sslkey=/etc/letsencrypt/privkey.pem sslrootcert=/etc/letsencrypt/root.crt" +psql "dbname=gis port=5432 user=docker host=localhost sslmode=verify-full sslcert=/etc/letsencrypt/fullchain.pem sslkey=/etc/letsencrypt/privkey.pem sslrootcert=/etc/letsencrypt/root.crt" ``` ## Postgres Replication Setup -The image supports replication out of the box. By default, replication is turned off. -The two main replication methods allowed are +The image supports replication out of the box. By default, replication is turned off. The two main +replication methods allowed are, + * Streaming replication * Logical replication - ### Database permissions and password authentication -Replication uses a dedicated user `REPLICATION_USER`. The role `${REPLICATION_USER}` -uses the default group role `pg_read_all_data`. You can read more about this -from the [PostgreSQL documentation](https://www.postgresql.org/docs/14/predefined-roles.html) -**Note:** When setting up replication you need to specify the password using the -environment variable `REPLICATION_PASS`. If you do not specify it a random strong -password will be generated. This is visible in the startup logs as well -as a text file within the container in `/tmp/REPLPASSWORD.txt`. +Replication uses a dedicated user `REPLICATION_USER`. The role `${REPLICATION_USER}` uses the +default group role `pg_read_all_data`. You can read more about this from the +[PostgreSQL documentation](https://www.postgresql.org/docs/14/predefined-roles.html) + +**Note:** When setting up replication you need to specify the password using the environment +variable `REPLICATION_PASS`. If you do not specify it a random strong password will be generated. +This is visible in the startup logs as well as a text file within the container in +`/tmp/REPLPASSWORD.txt`. ### Streaming replication -Replication allows you to maintain two or more synchronised copies of a database, with a -single **master** copy and one or more **replicant** copies. The animation below illustrates -this - the layer with the red boundary is accessed from the master database and the layer -with the green fill is accessed from the replicant database. When edits to the master -layer are saved, they are automatically propagated to the replicant. Note also that the -replicant is read-only. +Replication allows you to maintain two or more synchronized copies of a database, with a single +**master** copy and one or more **replicant** copies. The animation below illustrates this - the +layer with the red boundary is accessed from the master database and the layer with the green fill +is accessed from the replicant database. When edits to the master layer are saved, they are +automatically propagated to the replicant. Note also that the replicant is read-only. ```shell -docker run --name "streaming-replication" -e REPLICATION=true -e WAL_LEVEL='replica' -d -p 25432:5432 kartoza/postgis:14.3.2 +docker run --name "streaming-replication" -e REPLICATION=true -e WAL_LEVEL='replica' -d -p 25432:5432 kartoza/postgis:14.3.2 ``` -**Note** If you do not pass the env variable `REPLICATION_PASS` a random password -will be generated and will be visible from the logs or within the container in -`/tmp/REPLPASSWORD.txt` +**Note** If you do not pass the env variable `REPLICATION_PASS` a random password will be generated +and will be visible from the logs or within the container in `/tmp/REPLPASSWORD.txt` ![qgis](https://user-images.githubusercontent.com/178003/37755610-dd3b774a-2dae-11e8-9fa1-4877e2034675.gif) -This image is provided with replication abilities. We can -categorize an instance of the container as `master` or `replicant`. A `master` -instance means that a particular container has a role as a single point of -database write. A `replicant` instance means that a particular container will -mirror database content from a designated master. This replication scheme allows -us to sync databases. However, a `replicant` is only for read-only transaction, thus -we can't write new data to it. The whole database cluster will be replicated. +This image is provided with replication abilities. We can categorize an instance of the container +as `master` or `replicant`. A `master` instance means that a particular container has a role as a +single point of database write. A `replicant` instance means that a particular container will +mirror database content from a designated master. This replication scheme allows us to sync +databases. However, a `replicant` is only for read-only transaction, thus we can't write new data +to it. The whole database cluster will be replicated. #### Database permissions Since we are using a role `${REPLICATION_USER}`, we need to ensure that it has access to all -the tables in a particular schema. So if a user adds another schema called `data` -to the database `gis` he also has to update the permission for the user -with the following SQL assuming the `${REPLICATION_USER}` is called replicator +the tables in a particular schema. So if a user adds another schema called `data` to the database +`gis` he also has to update the permission for the user with the following SQL assuming the +`${REPLICATION_USER}` is called replicator, ```sql ALTER DEFAULT PRIVILEGES IN SCHEMA data GRANT SELECT ON TABLES TO replicator; ``` -**Note** You need to set up a strong password for replication otherwise the -default password for `${REPLICATION_USER}` will default to random generated string +**Note** You need to set up a strong password for replication otherwise the default password for +`${REPLICATION_USER}` will default to random generated string. -To experiment with the streaming replication abilities, you can see a [docker-compose.yml](replication_examples/replication/docker-compose.yml). -There are several environment variables that you can set, such as: +To experiment with the streaming replication abilities, you can see a +[docker-compose.yml](replication_examples/replication/docker-compose.yml). There are several +environment variables that you can set, such as: Master settings: + - **ALLOW_IP_RANGE**: A `pg_hba.conf` domain format which will allow specified host(s) - to connect into the container. This is needed to allow the `slave` to connect - into `master`, so specifically these settings should allow `slave` address. It is also needed to allow clients on other hosts to connect to either the slave or the master. + to connect into the container. This is needed to allow the `slave` to connect into `master`, so + specifically these settings should allow `slave` address. It is also needed to allow clients on + other hosts to connect to either the slave or the master. - **REPLICATION_USER** User to initiate streaming replication - **REPLICATION_PASS** Password for a user with streaming replication role Slave settings: + - **REPLICATE_FROM**: This should be the domain name or IP address of the `master` instance. It can be anything from the docker resolved name like that written in the sample, - or the IP address of the actual machine where you expose `master`. This is - useful to create cross machine replication, or cross stack/server. + or the IP address of the actual machine where you expose `master`. This is useful to create cross + machine replication, or cross stack/server. - **REPLICATE_PORT**: This should be the port number of `master` postgres instance. Will default to 5432 (default postgres port), if not specified. -- **DESTROY_DATABASE_ON_RESTART**: Default is `True`. Set to 'False' to prevent - this behaviour. A replicant will always destroy its current database on - restart, because it will try to sync again from `master` and avoid inconsistencies. +- **DESTROY_DATABASE_ON_RESTART**: Default is `True`. Set to 'False' to prevent this behavior. A + replicant will always destroy its current database on restart, because it will try to sync again + from `master` and avoid inconsistencies. - **PROMOTE_MASTER**: Default none. If set to any value then the current replicant - will be promoted to master. - In some cases when the `master` container has failed, we might want to use our `replicant` - as `master` for a while. However, the promoted replicant will break consistencies and - is not able to revert to replicant anymore, unless it is destroyed and resynced - with the new master. + will be promoted to master. In some cases when the `master` container has failed, we might want + to use our `replicant` as `master` for a while. However, the promoted replicant will break + consistencies and is not able to revert to replicant anymore, unless it is destroyed and + re-synced with the new master. - **REPLICATION_USER** User to initiate streaming replication - **REPLICATION_PASS** Password for a user with streaming replication role @@ -715,8 +711,8 @@ Do a manual image build by executing the `build.sh` script ./build.sh ``` -Go into the `replication_examples/streaming_replication` directory and experiment with the following Make -command to run both master and slave services. +Go into the `replication_examples/streaming_replication` directory and experiment with the +following `Make` command to run both master and slave services. ```shell make up @@ -739,10 +735,10 @@ You can try experiment with several scenarios to see how replication works #### Sync changes from master to replicant -You can use any postgres database tools to create new tables in master, by -connecting using `POSTGRES_USER` and `POSTGRES_PASS` credentials using exposed port. -In the streaming_replication example, the master database was exposed on port 7777. -Or you can do it via command line, by entering the shell: +You can use any postgres database tools to create new tables in master, by connecting using +`POSTGRES_USER` and `POSTGRES_PASS` credentials using exposed port. In the streaming_replication +example, the master database was exposed on port 7777. Or you can do it via command line, by +entering the shell: ```shell make master-shell @@ -750,10 +746,9 @@ make master-shell Then make any database changes using psql. -After that, you can see that the replicant follows the changes by inspecting the -slave database. You can, again, use database management tools using connection -credentials, hostname, and ports for replicant. Or you can do it via command line, -by entering the shell: +After that, you can see that the replicant follows the changes by inspecting the slave database. +You can, again, use database management tools using connection credentials, hostname, and ports for +replicant. Or you can do it via command line, by entering the shell: ```shell make node-shell @@ -763,25 +758,25 @@ Then view your changes using psql. #### Promoting replicant to master -You will notice that you cannot make changes in replicant, because it is read-only. -If somehow you want to promote it to master, you can specify `PROMOTE_MASTER: 'True'` -into slave environment and set `DESTROY_DATABASE_ON_RESTART: 'False'`. +You will notice that you cannot make changes in replicant, because it is read-only. If somehow you +want to promote it to master, you can specify `PROMOTE_MASTER: 'True'` into slave environment and +set `DESTROY_DATABASE_ON_RESTART: 'False'`. -After this, you can make changes to your replicant, but master and replicant will not -be in sync anymore. This is useful if the replicant needs to take over a failover master. -However, it is recommended to take additional action, such as creating a backup from the -slave so a dedicated master can be created again. +After this, you can make changes to your replicant, but master and replicant will not be in sync +anymore. This is useful if the replicant needs to take over a failover master. However, it is +recommended to take additional action, such as creating a backup from the slave so a dedicated +master can be created again. #### Preventing replicant database destroy on restart -You can optionally set `DESTROY_DATABASE_ON_RESTART: 'False'` after successful sync -to prevent the database from being destroyed on restart. With this setting you can -shut down your replicant and restart it later, and it will continue to sync using the existing -database (as long as there are no consistencies conflicts). +You can optionally set `DESTROY_DATABASE_ON_RESTART: 'False'` after successful sync to prevent the +database from being destroyed on restart. With this setting you can shut down your replicant and +restart it later, and it will continue to sync using the existing database (as long as there are no +consistencies conflicts). -However, you should note that this option doesn't mean anything if you didn't -persist your database volume. Because if it is not persisted, then it will be lost -on restart because docker will recreate the container. +However, you should note that this option doesn't mean anything if you didn't persist your database +volume. Because if it is not persisted, then it will be lost on restart because docker will recreate +the container. ### Logical replication @@ -797,18 +792,17 @@ For a detailed example see the docker-compose in the folder `replication_example ### Docker image versions -All instructions mentioned in the README are valid for the latest running image. -Other docker images might have a few missing features than the ones in the -latest image. We mainly do not back port changes to current stable images that are being -used in production. However, if you feel that some changes included -in the latest tagged version of the image are essential for the previous image -you can cherry-pick the changes against that specific branch and we will -test and merge. +All instructions mentioned in the README are valid for the latest running image. Other docker +images might have a few missing features than the ones in the latest image. We mainly do not back +port changes to current stable images that are being used in production. However, if you feel that +some changes included in the latest tagged version of the image are essential for the previous +image you can cherry-pick the changes against that specific branch and we will test and merge. ### Support -If you require more substantial assistance from [kartoza](https://kartoza.com) (because our work and interaction on docker-postgis is pro bono), -please consider taking out a [Support Level Agreeement](https://kartoza.com/en/shop/product/support) +If you require more substantial assistance from [kartoza](https://kartoza.com) (because our work +and interaction on docker-postgis is pro bono), please consider taking out a +[Support Level Agreement](https://kartoza.com/en/shop/product/support). ## Credits